Exactly Once Processing

Overview

Learn how Spark Structured Streaming achieves end-to-end exactly-once delivery guarantees using replayable sources, idempotent sinks, and transaction logs.

What You Will Learn

In this lesson, you will learn:

Exactly-Once Guarantees: Processing and delivering records exactly once.
Replayable Sources: Re-reading source events during crashes.
Idempotent Sinks: Discarding duplicate writes automatically.

Detailed Concept Explanation

"Exactly-Once" processing means that even if a system failure occurs, the final output results will reflect each input record being processed exactly once.

Spark achieves End-to-End Exactly-Once by combining three core mechanisms:

Replayable Source: The input source (like Kafka) must be able to replay messages from a specific offset if Spark needs to re-read them after a crash.
Idempotent Sink: The output destination (like Delta Lake) must be able to identify and discard duplicate writes if a micro-batch is re-executed after a failure.
Deterministic Logic: The transformations must produce the same output for a given input.

Code Examples

Python (PySpark) Implementation

python

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExactlyOnce").getOrCreate()

# Delta Lake serves as an idempotent sink out-of-the-box
stream = spark.readStream.format("rate").load()

query = stream.writeStream \
    .format("delta") \
    .option("checkpointLocation", "delta_checkpoints") \
    .start("delta_table")

query.awaitTermination(5)
query.stop()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram

SparkSession.builder

readStream

writeStream.format(delta).option(checkpointLocation)

start()

stop()

Scala Implementation

scala

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ExactlyOnceScala").getOrCreate()

val stream = spark.readStream.format("rate").load()

val query = stream.writeStream
  .format("delta")
  .option("checkpointLocation", "delta_checkpoints")
  .start("delta_table")

query.awaitTermination(5000)
query.stop()

Interview Perspective

What is required to achieve end-to-end exactly-once guarantees in Spark streaming?

To achieve end-to-end exactly-once guarantees, you need a replayable source (to re-fetch offsets during recovery), an idempotent sink (to safely re-write duplicate records during retries without duplicating data), and deterministic processing logic. Spark manages offsets in checkpoint logs to ensure coordination between the source and the sink.