advanced

Stream Joins

By DataPlayArena Technical Review BoardUpdated 2026-07-0910 min readRuntime Verified

Overview

Learn how to join streaming data with static datasets (Stream-to-Static) or combine two real-time streams (Stream-to-Stream).

What You Will Learn

In this lesson, you will learn:
  • Stream-to-Static Join: Enriching streams using lookup tables.
  • Stream-to-Stream Join: Combining two real-time events.
  • Watermark Joins: Preventing infinite state storage during stream joins.

Detailed Concept Explanation

Spark Structured Streaming supports joining streaming datasets.

Join Types

  1. Stream-to-Static Joins: Joins a streaming DataFrame with a static DataFrame. This is stateless and does not require watermarks. The static table is read once (or cached) and joined as stream rows arrive.
  2. Stream-to-Stream Joins: Joins two streaming DataFrames. This requires Spark to buffer events in memory (state store) to find matches. To prevent memory leaks, you must configure Watermarks and time-bound constraints on both streams.

Code Examples

Input Dataset Preview

Below is the static lookup table of user departments:

userIddept
1IT

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("StreamJoins").getOrCreate()

# Create static lookup DataFrame
static_users = spark.createDataFrame([(1, "IT")], ["userId", "dept"])

# Stream user events
events_stream = spark.readStream.format("rate").load() \
    .selectExpr("CAST(value AS INT) AS userId", "timestamp")

# Stream-to-Static Join
joined_stream = events_stream.join(static_users, "userId")

query = joined_stream.writeStream.format("console").start()
query.awaitTermination(5)
query.stop()

Expected Output

text
+------+-------------------+----+
|userId|          timestamp|dept|
+------+-------------------+----+
|     1|2026-07-09 19:30:00|  IT|
+------+-------------------+----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame(static)
readStream(events)
join(static_users
userId)
writeStream
start()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("StreamJoinsScala").getOrCreate()
import spark.implicits._

val staticUsers = Seq((1, "IT")).toDF("userId", "dept")

val eventsStream = spark.readStream.format("rate").load()
  .selectExpr("CAST(value AS INT) AS userId", "timestamp")

val joined = eventsStream.join(staticUsers, "userId")

val query = joined.writeStream.format("console").start()
query.awaitTermination(5000)
query.stop()

Expected Output

text
+------+-------------------+----+
|userId|          timestamp|dept|
+------+-------------------+----+
|     1|2026-07-09 19:30:00|  IT|
+------+-------------------+----+

Common Mistakes

  • Failing to Define Watermarks for Stream-to-Stream Joins: Attempting to join two streams without setting watermarks and time-bound constraints. Spark will keep all events in memory forever to check for matches, which will eventually crash the executors.

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.