intermediate
Output Modes
8 min readLast updated: 2026-07-09
Overview
Learn about the three output modes in Structured Streaming: Append, Complete, and Update, and when to use each.
What You Will Learn
In this lesson, you will learn:
- Append Mode: Writing new records only.
- Complete Mode: Overwriting the entire result table.
- Update Mode: Writing only rows changed since the last batch.
Detailed Concept Explanation
The Output Mode defines what data is written to the output sink when new records are processed in a micro-batch.
| Output Mode | Behavior | Use Case |
|---|---|---|
| append (Default) | Only new rows added since the last trigger are written to the sink. | Non-aggregating queries (simple maps/filters). |
| complete | The entire updated result table is rewritten to the sink. | Aggregations (e.g. running totals). |
| update | Only the rows in the result table that changed since the last trigger are written. | Aggregations where you only want to record changed keys. |
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("OutputModes").getOrCreate()
rate_stream = spark.readStream.format("rate").load()
grouped_counts = rate_stream.groupBy("value").count()
# Use Complete mode to output running aggregates
query = grouped_counts.writeStream \
.format("console") \
.outputMode("complete") \
.start()
query.awaitTermination(5)
query.stop()
Expected Output
text
+-----+-----+
|value|count|
+-----+-----+
| 0| 1|
| 1| 1|
+-----+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
readStream
groupBy(value).count()
writeStream.outputMode(complete)
start()
stop()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("OutputModesScala").getOrCreate()
val rateStream = spark.readStream.format("rate").load()
val groupedCounts = rateStream.groupBy("value").count()
val query = groupedCounts.writeStream
.format("console")
.outputMode("complete")
.start()
query.awaitTermination(5000)
query.stop()
Expected Output
text
+-----+-----+
|value|count|
+-----+-----+
| 0| 1|
| 1| 1|
+-----+-----+
Common Mistakes
- Applying Append Mode to Aggregations: Trying to use Append mode on aggregates without setting watermarks. Spark will throw an error because it cannot guarantee that keys won't receive more updates in future batches unless a watermark threshold is defined.