beginner

Introduction to Structured Streaming

By DataPlayArena Technical Review BoardUpdated 2026-07-098 min readRuntime Verified

Overview

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. Learn how it models streams as unbounded tables.

What You Will Learn

In this lesson, you will learn:
  • Unbounded Table Model: Treating streaming data as an append-only table.
  • Micro-Batch Processing: How Spark slices streams into small batches.
  • Declarative API: Reusing DataFrame batch queries for real-time streams.

Detailed Concept Explanation

Structured Streaming models streaming data as an unbounded, continuously growing table. Every incoming record is treated as a new row appended to this table.

text
Stream Data  === (new data) ===>  Unbounded Table  === (incremental query) ===> Result Table

Key Properties

  • Micro-batch execution: By default, Spark groups stream data into micro-batches and runs them through the execution engine.
  • Unified API: The same DataFrame queries, transformations, and SQL statements you run on static data run identically on streaming data.

Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("StreamingBasics").getOrCreate()

# Read streaming text data from socket connection port 9999
lines_df = spark.readStream.format("socket") \
    .option("host", "localhost") \
    .option("port", 9999) \
    .load()

# Print stream status
print("Is streaming active:", lines_df.isStreaming)

Expected Output

text
Is streaming active: True

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
readStream.format(socket)
load()
isStreaming

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("StreamingBasicsScala").getOrCreate()

val linesDF = spark.readStream.format("socket")
  .option("host", "localhost")
  .option("port", 9999)
  .load()

println(s"Is streaming active: ${linesDF.isStreaming}")

Expected Output

text
Is streaming active: true

SQL Perspective

SQL Query Support:

To run SQL queries on a stream, you must first register the streaming DataFrame as a temporary view:

python
lines_df.createOrReplaceTempView("realtime_lines")
# Any downstream query reads the view as a stream

Common Mistakes

  • Using Batch Reader Methods: Using spark.read instead of spark.readStream. The standard read method reads files statically at a single moment, whereas readStream sets up a continuous query listener.

Best Practices

  • Use socket only for testing: Do not use the Socket source in production; it is not fault-tolerant and does not support recovery.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.