intermediate
Output Modes
Overview
Learn about the three output modes in Structured Streaming: Append, Complete, and Update, and when to use each.
What You Will Learn
In this lesson, you will learn:
- Append Mode: Writing new records only.
- Complete Mode: Overwriting the entire result table.
- Update Mode: Writing only rows changed since the last batch.
Detailed Concept Explanation
The Output Mode defines what data is written to the output sink when new records are processed in a micro-batch.
| Output Mode | Behavior | Use Case |
|---|---|---|
| append (Default) | Only new rows added since the last trigger are written to the sink. | Non-aggregating queries (simple maps/filters). |
| complete | The entire updated result table is rewritten to the sink. | Aggregations (e.g. running totals). |
| update | Only the rows in the result table that changed since the last trigger are written. | Aggregations where you only want to record changed keys. |
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("OutputModes").getOrCreate()
rate_stream = spark.readStream.format("rate").load()
grouped_counts = rate_stream.groupBy("value").count()
# Use Complete mode to output running aggregates
query = grouped_counts.writeStream \
.format("console") \
.outputMode("complete") \
.start()
query.awaitTermination(5)
query.stop()
Expected Output
text
+-----+-----+
|value|count|
+-----+-----+
| 0| 1|
| 1| 1|
+-----+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
readStream
groupBy(value).count()
writeStream.outputMode(complete)
start()
stop()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("OutputModesScala").getOrCreate()
val rateStream = spark.readStream.format("rate").load()
val groupedCounts = rateStream.groupBy("value").count()
val query = groupedCounts.writeStream
.format("console")
.outputMode("complete")
.start()
query.awaitTermination(5000)
query.stop()
Expected Output
text
+-----+-----+
|value|count|
+-----+-----+
| 0| 1|
| 1| 1|
+-----+-----+
Common Mistakes
- Applying Append Mode to Aggregations: Trying to use Append mode on aggregates without setting watermarks. Spark will throw an error because it cannot guarantee that keys won't receive more updates in future batches unless a watermark threshold is defined.
Related Topics
DPA
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.