Windowed Aggregations
Overview
Learn how to aggregate streaming data over time intervals using Tumbling (non-overlapping) and Sliding (overlapping) window queries.
What You Will Learn
In this lesson, you will learn:
- Tumbling Windows: Fixed, contiguous time ranges.
- Sliding Windows: Overlapping time intervals.
- Window Aggregations: Calculating metrics per time window.
Detailed Concept Explanation
Structured Streaming supports aggregations over time-based windows, grouping events based on when they occurred.
Window Types
- Tumbling Windows: Fixed-size, non-overlapping time intervals (e.g. every 10 minutes).
- Sliding Windows: Fixed-size, overlapping time intervals (e.g. 10-minute windows sliding every 5 minutes). A single event can fall into multiple sliding windows.
Code Examples
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import window
spark = SparkSession.builder.appName("Windowing").getOrCreate()
# Stream sensor data
sensor_stream = spark.readStream.format("rate").load()
# Calculate sliding window counts
windowed_counts = sensor_stream.groupBy(
window(sensor_stream.timestamp, "10 minutes", "5 minutes")
).count()
query = windowed_counts.writeStream \
.format("console") \
.outputMode("complete") \
.start()
query.awaitTermination(5)
query.stop()
Expected Output
+------------------------------------------+-----+
| window|count|
+------------------------------------------+-----+
|[2026-07-09 19:20:00, 2026-07-09 19:30:00]| 10|
+------------------------------------------+-----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.window
val spark = SparkSession.builder().appName("WindowingScala").getOrCreate()
val sensorStream = spark.readStream.format("rate").load()
val windowedCounts = sensorStream.groupBy(
window($"timestamp", "10 minutes", "5 minutes")
).count()
val query = windowedCounts.writeStream
.format("console")
.outputMode("complete")
.start()
query.awaitTermination(5000)
query.stop()
Expected Output
+------------------------------------------+-----+
| window|count|
+------------------------------------------+-----+
|[2026-07-09 19:20:00, 2026-07-09 19:30:00]| 10|
+------------------------------------------+-----+
Common Mistakes
- Failing to use Complete/Update Mode: Running windowed aggregations with the default Append mode without setting a watermark. Grouped aggregations must use Complete or Update mode to output incremental changes.
Interview Perspective
What is the difference between tumbling and sliding windows in Spark?
Tumbling windows are contiguous, fixed-size, and non-overlapping. For example, a 10-minute window will group data from 12:00-12:10, 12:10-12:20, etc. Sliding windows are fixed-size but overlap based on a slide interval. For example, a 10-minute window sliding every 5 minutes will group data from 12:00-12:10, 12:05-12:15, etc., meaning a single record can fall into multiple windows.
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.