Reading Streams

Overview

Learn how to configure Spark Structured Streaming to read data from diverse sources including directories, network sockets, and message brokers.

What You Will Learn

Detailed Concept Explanation

To read streaming data, you use spark.readStream instead of the batch spark.read.

Common streaming sources:

1. rate: A mock source that generates data at a specified number of rows per second, used for testing.
2. socket: Reads text lines from a TCP socket port.
3. file: Reads files written in a directory as a stream. Requires defining an explicit schema.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ReadStream").getOrCreate()

# Create streaming source that generates 5 rows/sec
rate_df = spark.readStream.format("rate") \
    .option("rowsPerSecond", 5) \
    .load()

# Check structure schema
rate_df.printSchema()

Expected Output

root
 |-- timestamp: timestamp (nullable = true)
 |-- value: long (nullable = true)

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ReadStreamScala").getOrCreate()

val rateDF = spark.readStream.format("rate")
  .option("rowsPerSecond", 5)
  .load()

rateDF.printSchema()

Expected Output

root
 |-- timestamp: timestamp (nullable = true)
 |-- value: long (nullable = true)

Common Mistakes

• Omitting schema for file streams: Trying to read files (like JSON or CSV) using readStream without setting a schema. Spark requires a schema up-front to construct the streaming query, otherwise it will throw an AnalysisException.

Interactive Challenges

Related Topics

• Writing Streams
• Kafka Streaming
• File Streaming