Introduction to Structured Streaming
Overview
Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. Learn how it models streams as unbounded tables.
What You Will Learn
In this lesson, you will learn:
- Unbounded Table Model: Treating streaming data as an append-only table.
- Micro-Batch Processing: How Spark slices streams into small batches.
- Declarative API: Reusing DataFrame batch queries for real-time streams.
Detailed Concept Explanation
Structured Streaming models streaming data as an unbounded, continuously growing table. Every incoming record is treated as a new row appended to this table.
Stream Data === (new data) ===> Unbounded Table === (incremental query) ===> Result Table
Key Properties
- Micro-batch execution: By default, Spark groups stream data into micro-batches and runs them through the execution engine.
- Unified API: The same DataFrame queries, transformations, and SQL statements you run on static data run identically on streaming data.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("StreamingBasics").getOrCreate()
# Read streaming text data from socket connection port 9999
lines_df = spark.readStream.format("socket") \
.option("host", "localhost") \
.option("port", 9999) \
.load()
# Print stream status
print("Is streaming active:", lines_df.isStreaming)
Expected Output
Is streaming active: True
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("StreamingBasicsScala").getOrCreate()
val linesDF = spark.readStream.format("socket")
.option("host", "localhost")
.option("port", 9999)
.load()
println(s"Is streaming active: ${linesDF.isStreaming}")
Expected Output
Is streaming active: true
SQL Perspective
SQL Query Support:
To run SQL queries on a stream, you must first register the streaming DataFrame as a temporary view:
lines_df.createOrReplaceTempView("realtime_lines")
# Any downstream query reads the view as a stream
Common Mistakes
- Using Batch Reader Methods: Using
spark.readinstead ofspark.readStream. The standardreadmethod reads files statically at a single moment, whereasreadStreamsets up a continuous query listener.
Best Practices
- Use socket only for testing: Do not use the Socket source in production; it is not fault-tolerant and does not support recovery.
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.