SnappyData Monitoring Console¶

SnappyData Monitoring Console is a dashboard that provides a real-time view into cluster members, member logs, resource usage, running Jobs, SQL queries along with performance data. This simple widget based view allows you to navigate easily, visualize, and monitor your cluster. You can monitor the overall status of the cluster as well as the status of each member in the cluster. All the usage details are automatically refreshed after every five seconds.

To access SnappyData Monitoring Console, start your cluster and open http:<leadhost>:5050/dashboard/ in the web browser.

The following topics are covered in this section:

• Dashboard

• Member Details

• Jobs

• Structured Streaming

• Stages

• Spark Cache

• Environment

• Executors

• SQL

In cases where you cannot access the SnappyData Monitoring Console to analyse Jobs and tasks, you must turn on the Spark History server.

On the top-right side of the SnappyData Monitoring Console page, you can view the version details of SnappyData Snapshot. When you click this, the name and version of the product, the build details, the source revision details and the version number of the underlying spark are displayed.

On the top of the dashboard, the date and time details about when the cluster is started are displayed. The period till when the cluster is running is also shown.

Dashboard¶

The Dashboard page graphically presents various cluster-level statistics that can be used to monitor the current health status of a cluster. The statistics on the dashboard page can be set to update automatically after every five seconds. If you want to turn off the auto-refresh, use the Auto Refresh switch that is provided on the upper-right corner.

The Dashboard page displays the following sections:

• Cluster

• Members

• Tables

• External Tables

You can use the search and sort functionalities in any of the sections, except for the Cluster section. Sorting is enabled to sort items in an ascending and descending order. Further, you can also set the number of items that must be listed in each of these sections.

Cluster¶

In the Cluster section, you can view the following graphs which are automatically refreshed:

On the Dashboard page, the Cluster section displays the date and time when the cluster was launched. It also displays the cluster's up-time since cluster was launched.

You can view the total number of physical CPU cores present in your cluster on the top-right side of the page.

Members¶

In the Members section, you can view, in a tabular format, the details of each locator, data server, and lead member within a cluster. The details are automatically refreshed after every five seconds.

This table provides member details in the following columns:

Status	Description
	Member is running.
	Member has stopped or is unavailable.

Tables¶

The Tables section lists all the tables in the cluster along with their corresponding statistical details. All these details are automatically refreshed after every five seconds.

The following columns are displayed in this section:

External Tables¶

The External Tables section lists all the external tables present in the cluster along with their various statistical details. The displayed details are automatically refreshed after every five seconds.

The following columns are displayed in this section:

Member Details¶

The Member Details view shows the usage trend and statistics of a specific cluster member. To check the Member Details view, go to the Members section and click the link in the Member column. Here you can also view the Member Logs generated for a cluster member. The usage trends and the statistics of a specific member are auto updated periodically after every five seconds. If you want to turn off the auto-refresh, use the Auto Refresh switch that is provided on the upper-right corner. You can view, on demand, the latest logs by clicking on the Load New button provided at the bottom of the logs. You can also click the Load More button to view the older logs.

Member Statistics¶

The following member specific statistics are displayed:

The usage trends of the member are represented in the following graphs:

Member Logs¶

In the Member Details page, you can view the logs generated for a single member in the cluster.

The following details are included:

SQL¶

The SQL section shows all the queries and their corresponding details along with their execution plans and stagewise breakups.

Spark Cache¶

Spark Cache is the inbuilt storage mechanism of Spark. When you do a dataSet.cache(), it uses this storage to store the dataset's data in a columnar format. This storage can be configured to be one of the following:

• MEMORY_ONLY,
• MEMORY_AND_DISK,
• MEMORY_ONLY_SER,
• MEMORY_AND_DISK_SER,
• DISK_ONLY,
• MEMORY_ONLY_2,
• MEMORY_AND_DISK_2

For more details, see RDD Persistence section.

Environment¶

The Environment page provides detailed configurations for Spark environment including JVM, SparkContext, and SparkSession.

Executors¶

Executors are the entities that perform the tasks within a Spark job. Each Spark job is divided into multiple stages which can have one or more tasks depending on the number of partitions to be processed. All these tasks are scheduled on executor nodes which actually run them.

Jobs¶

The Jobs page lists all the Spark jobs. Each Spark action is translated as a Spark job. A job encapsulates the whole execution of an API or SQL. For example, dataSet.count() triggers a job.

• Status: Displays the status of the job.

• Stages: Click on the stage to view its details. The table displays the time taken for the completion of each stage.

Stages¶

The Stages page displays the stage details of a Spark Job. Each Spark job is segregated into one or more stages. Each stage is an execution boundary where data exchange between nodes is required.

On this page, you can view the total time required for all the tasks in a job to complete. You can also view if any of the tasks got delayed for completion. This may occur in case of uneven data distribution.

• Scheduler Delay indicates the waiting period for the task. Delays can be caused if there are too many concurrent jobs.

• Shuffle reads and writes: Shuffles are written to disk and can take a lot of time to write and read. This can be avoided by using colocated and replicated tables. You can use high-performance SSD drives for temporary storage (spark.local.dir) to improve shuffle time.

• Number of parallel tasks: Due to concurrency, multiple queries may take cores and a specific query can take longer. To fix this, you can create a new scheduler and assign appropriate cores to it.

• GC time: Occasionally, on-heap object creation can slow down a query because of garbage collection. In these cases, it is recommended that you increase the on-heap memory, especially when you have row tables.

Structured Streaming¶

Structured Streaming tab provides a real-time view into all the running structured streaming queries in the cluster. You can navigate, visualize, and monitor all the structured streaming queries from a single location. Further, you can also track various statistics of these queries as listed here. All these statistics are automatically refreshed after every five seconds.

You can view the Structured Streaming tab only for the monitoring console of the corresponding cluster. For example, if you submit a streaming query on the embedded cluster using snappy-job, then the Structured Streaming tab is available on the monitoring console of the embedded cluster. If you submit a streaming query as a Smart Connector job, on a separate spark cluster, then the Structured Streaming tab is available on the monitoring console of the Spark cluster where the Smart Connector job is submitted.

The following details are shown on the Structured Streaming tab:

Query Statistics Information¶

Structured Streaming Queries Sources¶

Structured Streaming Queries Sink¶

Graphical Charts¶

The Structured Streaming tab presents four types of graphical charts. All these charts display historical trends or behavior of respective parameters for a given number of data points or sample size. You can customize the data points sample size by setting the configuration parameter spark.sql.streaming.uiTrendsMaxSampleSize. The default value is 60 data points, which indicates that all the graphical charts display data of the last 60 events.

Input Records¶

This line chart displays the number of input records received for each of the batches in the last n trigger events.

Input Rate Vs. Processing Rate¶

This line chart displays the behavior of two parameters, for the last n triggers, in two curve lines. n refers to value configured for spark.sql.streaming.uiTrendsMaxSampleSize, which defaults to 60. One line represents the input rate that is the number of input records per second in the given triggers. The other line represents the processing rate, which is the number of records processed per second in the given triggers. Processing rate should ideally remain higher than Input rate. If it is below the input rate, then that indicates that the streaming query is falling behind the input data and you may need to add more resources to the cluster to keep up the processing speed.

Processing Time (in Milliseconds)¶

This line chart displays the time taken to process all the input records received in each of the batches for the last n triggers. This chart also displays the processing threshold line, which is the expected maximum time limit in milliseconds for processing a batch. This chart allows you to supervise whether the processing is happening within the time limit specified for Event Trigger Interval. In case the trend line for the Processing Time trend line is continuously below the Processing Threshold, then that indicates that there are enough resources to handle the streaming query operations. However, if the Processing Time trend line is often above the Processing Threshold, then that indicates that resources are insufficient to handle the streaming query operations, and you must add resources to manage the streaming input load. The Processing Threshold line will be always at zero if the trigger interval is not specified while defining the query.

Aggregation State¶

This line chart is displayed only if the streaming query is using one or more aggregate functions.

SQL¶

The SQL section shows all the queries and their corresponding details along with their execution plans and stagewise breakups.

Spark History Server¶

The Spark History server is a Spark UI extension. It is a web application that lets you analyze the running as well as completed SQL queries and the associated Spark jobs. The metadata collected in the form of event logs by the Spark History server can be shipped from a SnappyData cluster to another SnappyData cluster for further analysis.

The first step in tuning query performance in SnappyData is to understand the query physical plan that is available through the SQL tab on the SnappyData Monitoring console. The detailed execution plan requires one to understand the jobs and tasks associated with the query. This is available in the Jobs/Tasks tab. However, if the SnappyData Monitoring console is not accessible to the investigator, it becomes a difficult exercise. To overcome this, it is recommended to turn on the History server for production applications.

To turn on the History server, do the following:

1. Ensure to provide a shared disk that can be accessed from all the SnappyData nodes. If you do not have the NFS access, use HDFS. Provide the necessary permissions to access a shared folder when you start SnappyData.

2. Enable event logging for the Spark jobs. For example, specify the following properties in conf/lead:

       -spark.eventLog.enabled=true -spark.eventLog.dir=hdfs://namenode/shared/spark-logs
   

3. Additionally, set the following property in conf/spark-defaults.conf file (it defaults to /tmp/spark-events):

       spark.history.fs.logDirectory      hdfs://namenode/shared/spark-logs
   

   If this property is not set and the directory /tmp/spark-events does not exist, Spark History server fails to start.

4. By default, the history server loads only 1000 most recent events (SQL, Jobs, Stages). Older events will not be visible. To retain more number of events, set additional properties using SPARK_HISTORY_OPTS, and also increase the heap-size of the history server as follows in the file conf/SPARK_ENV.sh.

        SPARK_HISTORY_OPTS="-Dspark.history.fs.numReplayThreads=16 -Dspark.ui.retainedJobs=10000 -Dspark.ui.retainedStages=20000 -Dspark.sql.ui.retainedExecutions=10000"
        SPARK_DAEMON_MEMORY=12g
   

   Note

   When you retain a high number of events, it increases the disk space requirement. Therefore, ensure that the disk store, where the events are stored, has adequate disk space.

5. Start the History server.

       ./sbin/start-history-server.sh
   

   This creates a web interface at http://:18080 by default, listing incomplete and completed instances of SQL queries and the associated Spark jobs and tasks. For more details about History server, refer to Configuring History Server.