Web UI & Monitoring
Overview
Spark's Web UI is the primary debugging environment for checking stage execution times, partition skew, task garbage collection overheads, and active executors.
What You Will Learn
In this lesson, you will learn:
- Accessing Web UI: How to locate the diagnostic portal in local and cluster runs.
- Jobs and Stages Tabs: How to trace execution times and DAG shapes.
- Storage Tab: How to monitor the size of cached datasets.
- Diagnostics: How to identify data skew and garbage collection bottlenecks.
Detailed Concept Explanation
Accessing the Web UI
Whenever you start a SparkSession locally, Spark launches an internal web diagnostic server, usually accessible at:
http://localhost:4040
If multiple sessions run concurrently on the same machine, Spark increments the port index (e.g., 4041, 4042).
Key Diagnostics Tabs
1. Jobs Tab
Shows the status of all active and completed jobs. A new "Job" is created every time your code triggers an Action (like .show() or .collect()).
2. Stages Tab
Details the stages within each job. This is where you diagnose Data Skew:
- If the Max task execution time is much larger than the Median time, some tasks are processing much more data than others, indicating skewed keys.
- Shows Shuffle Read/Write volumes.
3. Storage Tab
Lists all RDDs and DataFrames that have been cached in memory. It displays their memory size, disk footprint, and what percentage of partitions are successfully stored in RAM.
4. Executors Tab
Monitors CPU core usage, memory footprint, and Garbage Collection (GC) Time for each executor. If GC Time exceeds 10% of total compute time, you need to adjust your memory settings or partition count.
Common Mistakes
- Closing Session Early: Once
spark.stop()is called, the local Web UI server shuts down. You cannot access it after your job finishes unless a persistent History Server is configured.
Best Practices
- Clean Storage: Call
.unpersist()to free up RAM in the Storage tab when a cached DataFrame is no longer needed. - Check Task Counts: Keep task counts balanced—too many tasks create high scheduling overhead, while too few tasks leave CPU cores idle.
Interview Perspective
How do you identify data skew using the Spark Web UI?
Navigate to the Stages tab and look at the summary metrics table for the tasks in a stage. If there is a large gap between the 75th percentile and the Max task execution time, or if the Max Shuffle Read size is gigabytes while the Median is megabytes, it means a few tasks are doing most of the work, indicating data skew.