Monitoring & Debugging

Overview

Learn how to monitor running applications, inspect the History Server, and troubleshoot failed driver and executor runs.

What You Will Learn

Detailed Concept Explanation

Troubleshooting failed or slow distributed Spark applications requires checking logs across different systems.

Monitoring Tools

1. Spark Web UI (Port 4040): Available while the driver application is active. Shows active stages, task durations, cached memory states, and running DAGs.
2. Spark History Server (Port 18080): Displays execution details for completed jobs. Requires configuring Spark to write event log files to shared storage (like HDFS or S3).
3. Driver Logs: Check the driver logs to troubleshoot query compile errors, driver out-of-memory errors, and configuration issues.
4. Executor Logs (stdout/stderr): Check executor logs on worker nodes to troubleshoot task failures, runtime data errors, and executor crashes.

Code Examples

Event Logging Configuration (to enable History Server access)

# Properties configuration defaults inside spark-defaults.conf
spark.eventLog.enabled           true
spark.eventLog.dir               hdfs://namenode:8020/spark-history-logs
spark.history.fs.logDirectory    hdfs://namenode:8020/spark-history-logs

Execution Plan Diagram (Python & Scala)

Related Topics

• Web UI & Monitoring
• Logging
• Memory Management