intermediate

Checkpointing

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how to configure checkpointing directories in Spark Structured Streaming to save metadata and offsets for query recovery.

What You Will Learn

In this lesson, you will learn:
  • Checkpoint Directory: Storing streaming metadata and states.
  • Recovery Mechanism: Restoring query runs after failures.
  • Configuration Options: Setting checkpointLocation options.

Detailed Concept Explanation

A streaming query must be resilient to failures. If an executor or driver node crashes, Spark must be able to restart and resume processing exactly where it left off.

To achieve this, you configure a Checkpoint Directory on durable storage (like HDFS or S3).

Spark writes query state details to this folder:

  • Offsets: The read markers representing which records have been read from the source.
  • Commit Log: The list of micro-batches successfully processed.
  • State Store files: The running aggregation states.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("Checkpointing").getOrCreate()

rate_stream = spark.readStream.format("rate").load()

# Write to storage with checkpointing
query = rate_stream.writeStream \
    .format("parquet") \
    .option("path", "output_data") \
    .option("checkpointLocation", "checkpoints/query1") \
    .start()

query.awaitTermination(5)
query.stop()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream
writeStream.format(parquet).option(checkpointLocation)
start()
stop()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CheckpointingScala").getOrCreate()

val rateStream = spark.readStream.format("rate").load()

val query = rateStream.writeStream
  .format("parquet")
  .option("path", "output_data")
  .option("checkpointLocation", "checkpoints/query1")
  .start()

query.awaitTermination(5000)
query.stop()

Common Mistakes

  • Sharing checkpoint paths: Reusing the same checkpoint directory path for multiple different queries. This corrupts the metadata offsets and crashes the query. Every streaming query must have its own unique checkpoint folder.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.