Streaming Architecture

Overview

Explore the internal architecture of Spark Structured Streaming, including the role of catalogs, logs, and metadata checkpoints.

What You Will Learn

In this lesson, you will learn:

Catalyst Integration: How the query optimizer compiles streaming plans.
Write-Ahead Logs (WAL): Guaranteeing offset state recoveries.
State Stores: Storing aggregations in memory and on disk.

Detailed Concept Explanation

Structured Streaming integrates directly with Spark's core SQL engine, leveraging the Catalyst Optimizer to construct logical and physical query plans.

Key Components

Source: An input stream (e.g. Apache Kafka or directory files) that must be replayable, allowing Spark to re-read data during crashes.
Write-Ahead Log (WAL): Spark writes incoming metadata offsets to a secure Write-Ahead Log in the checkpoint directory before processing a micro-batch, ensuring that no offsets are lost if a crash occurs.
State Store: A key-value database (like HDFSBackedStateStore or RocksDB) that tracks running aggregates (like group totals) across micro-batch steps.
Sink: The destination (e.g. Delta Lake or Kafka) where processed rows are written. Sinks must be idempotent to ensure exactly-once delivery.

Best Practices

Use RocksDB for Large States: When tracking millions of keys in stateful processing, configure RocksDB as the state store to avoid executor out-of-memory errors: spark.conf.set("spark.sql.streaming.stateStore.providerClass", "org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider")

Interview Perspective

How does Structured Streaming achieve fault tolerance?

Spark Structured Streaming achieves fault tolerance through a combination of replayable sources, checkpoint metadata files (including write-ahead logs), and idempotent sinks. If a worker node or driver fails, the engine restarts, reads the WAL to retrieve the last processed offsets, reconstructs the query state from the state store, and re-executes the batch.