advanced

Exactly Once Processing

By DataPlayArena Technical Review BoardUpdated 2026-07-0910 min readRuntime Verified

Overview

Learn how Spark Structured Streaming achieves end-to-end exactly-once delivery guarantees using replayable sources, idempotent sinks, and transaction logs.

What You Will Learn

In this lesson, you will learn:
  • Exactly-Once Guarantees: Processing and delivering records exactly once.
  • Replayable Sources: Re-reading source events during crashes.
  • Idempotent Sinks: Discarding duplicate writes automatically.

Detailed Concept Explanation

"Exactly-Once" processing means that even if a system failure occurs, the final output results will reflect each input record being processed exactly once.

Spark achieves End-to-End Exactly-Once by combining three core mechanisms:

  1. Replayable Source: The input source (like Kafka) must be able to replay messages from a specific offset if Spark needs to re-read them after a crash.
  2. Idempotent Sink: The output destination (like Delta Lake) must be able to identify and discard duplicate writes if a micro-batch is re-executed after a failure.
  3. Deterministic Logic: The transformations must produce the same output for a given input.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ExactlyOnce").getOrCreate()

# Delta Lake serves as an idempotent sink out-of-the-box
stream = spark.readStream.format("rate").load()

query = stream.writeStream \
    .format("delta") \
    .option("checkpointLocation", "delta_checkpoints") \
    .start("delta_table")

query.awaitTermination(5)
query.stop()

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream
writeStream.format(delta).option(checkpointLocation)
start()
stop()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ExactlyOnceScala").getOrCreate()

val stream = spark.readStream.format("rate").load()

val query = stream.writeStream
  .format("delta")
  .option("checkpointLocation", "delta_checkpoints")
  .start("delta_table")

query.awaitTermination(5000)
query.stop()

Interview Perspective

What is required to achieve end-to-end exactly-once guarantees in Spark streaming?

To achieve end-to-end exactly-once guarantees, you need a replayable source (to re-fetch offsets during recovery), an idempotent sink (to safely re-write duplicate records during retries without duplicating data), and deterministic processing logic. Spark manages offsets in checkpoint logs to ensure coordination between the source and the sink.


Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.