intermediate

Output Modes

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn about the three output modes in Structured Streaming: Append, Complete, and Update, and when to use each.

What You Will Learn

In this lesson, you will learn:
  • Append Mode: Writing new records only.
  • Complete Mode: Overwriting the entire result table.
  • Update Mode: Writing only rows changed since the last batch.

Detailed Concept Explanation

The Output Mode defines what data is written to the output sink when new records are processed in a micro-batch.

Output ModeBehaviorUse Case
append (Default)Only new rows added since the last trigger are written to the sink.Non-aggregating queries (simple maps/filters).
completeThe entire updated result table is rewritten to the sink.Aggregations (e.g. running totals).
updateOnly the rows in the result table that changed since the last trigger are written.Aggregations where you only want to record changed keys.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("OutputModes").getOrCreate()

rate_stream = spark.readStream.format("rate").load()
grouped_counts = rate_stream.groupBy("value").count()

# Use Complete mode to output running aggregates
query = grouped_counts.writeStream \
    .format("console") \
    .outputMode("complete") \
    .start()

query.awaitTermination(5)
query.stop()

Expected Output

text
+-----+-----+
|value|count|
+-----+-----+
|    0|    1|
|    1|    1|
+-----+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream
groupBy(value).count()
writeStream.outputMode(complete)
start()
stop()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("OutputModesScala").getOrCreate()

val rateStream = spark.readStream.format("rate").load()
val groupedCounts = rateStream.groupBy("value").count()

val query = groupedCounts.writeStream
  .format("console")
  .outputMode("complete")
  .start()

query.awaitTermination(5000)
query.stop()

Expected Output

text
+-----+-----+
|value|count|
+-----+-----+
|    0|    1|
|    1|    1|
+-----+-----+

Common Mistakes

  • Applying Append Mode to Aggregations: Trying to use Append mode on aggregates without setting watermarks. Spark will throw an error because it cannot guarantee that keys won't receive more updates in future batches unless a watermark threshold is defined.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.