Introduction to Structured Streaming

Overview

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. Learn how it models streams as unbounded tables.

What You Will Learn

Detailed Concept Explanation

Structured Streaming models streaming data as an unbounded, continuously growing table. Every incoming record is treated as a new row appended to this table.

Stream Data  === (new data) ===>  Unbounded Table  === (incremental query) ===> Result Table

Key Properties

• Micro-batch execution: By default, Spark groups stream data into micro-batches and runs them through the execution engine.
• Unified API: The same DataFrame queries, transformations, and SQL statements you run on static data run identically on streaming data.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("StreamingBasics").getOrCreate()

# Read streaming text data from socket connection port 9999
lines_df = spark.readStream.format("socket") \
    .option("host", "localhost") \
    .option("port", 9999) \
    .load()

# Print stream status
print("Is streaming active:", lines_df.isStreaming)

Expected Output

Is streaming active: True

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("StreamingBasicsScala").getOrCreate()

val linesDF = spark.readStream.format("socket")
  .option("host", "localhost")
  .option("port", 9999)
  .load()

println(s"Is streaming active: ${linesDF.isStreaming}")

Expected Output

Is streaming active: true

SQL Perspective

lines_df.createOrReplaceTempView("realtime_lines")
# Any downstream query reads the view as a stream

Common Mistakes

• Using Batch Reader Methods: Using spark.read instead of spark.readStream. The standard read method reads files statically at a single moment, whereas readStream sets up a continuous query listener.

Best Practices

• Use socket only for testing: Do not use the Socket source in production; it is not fault-tolerant and does not support recovery.

Related Topics

• Reading Streams
• Writing Streams
• Streaming Architecture