beginner

Writing Streams

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Learn how to define streaming outputs (sinks), configure output modes, and start streaming queries.

What You Will Learn

In this lesson, you will learn:
  • writeStream API: Setting output configurations.
  • Console Sink: Writing streams to stdout for debugging.
  • Query Lifecycle: Starting queries and waiting for termination.

Detailed Concept Explanation

A streaming query runs in the background. To start the query, you call writeStream on a streaming DataFrame, specify the destination format, and call .start().

Common Sinks

  • console: Prints outputs to stdout, used for debugging.
  • memory: Saves records into an in-memory table that can be queried using standard SQL.
  • parquet / csv / json: Saves records directly to files on disk. Requires checkpointing.

To prevent your driver script from exiting immediately after starting the query, call .awaitTermination().


Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("WriteStream").getOrCreate()

# Create rate source stream
rate_df = spark.readStream.format("rate").load()

# Start streaming query and output to console
query = rate_df.writeStream \
    .format("console") \
    .outputMode("append") \
    .start()

# Wait 5 seconds, then stop query
query.awaitTermination(5)
query.stop()

Expected Output

text
+-------------------+-----+
|          timestamp|value|
+-------------------+-----+
|2026-07-09 19:30:00|    1|
+-------------------+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream.format(rate)
load()
writeStream.format(console)
outputMode(append)
start()
awaitTermination(5)
stop()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("WriteStreamScala").getOrCreate()

val rateDF = spark.readStream.format("rate").load()

val query = rateDF.writeStream
  .format("console")
  .outputMode("append")
  .start()

query.awaitTermination(5000)
query.stop()

Expected Output

text
+-------------------+-----+
|          timestamp|value|
+-------------------+-----+
|2026-07-09 19:30:00|    1|
+-------------------+-----+

Common Mistakes

  • Forgetting to Call start(): Calling .writeStream.format("console") without calling .start(). Spark will not initialize the query or process any data.

Best Practices

  • Always specify checkpointLocation: Except for console and memory sinks, always set checkpointLocation when writing to storage to enable query recovery.

Interactive Challenges

Challenge 1: Stop query (Beginner)

Which method is called on a active StreamingQuery object to stop execution?


Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.