Interview Focus: StreamingDifficulty: Hard

Streaming Order Processing

Estimated duration: 20 mins

Lab Overview & Scenario

Read streaming order feeds and filter fraudulent purchases in real time.

Business Scenario

The security team wants to detect fraud as orders happen. You need to write a streaming query that processes real-time sales streams, filters out orders with values greater than $5,000, and writes them to a sink.

1. Local Raw Mock Dataset

The processing pipeline will ingest the following local mock file structures (e.g. simulated as `dataset.csv`):

order_id	card_id	amount
5001	9001	150.0
5002	9002	6000.0
5003	9001	450.0
5004	9003	5500.0
5005	9002	120.0

View Raw CSV Dataset

order_id,card_id,amount
5001,9001,150.0
5002,9002,6000.0
5003,9001,450.0
5004,9003,5500.0
5005,9002,120.0

2. Your Tasks

Simulate a streaming read from a source using readStream (simulated).

Filter out orders with amount greater than 5000.

Write the output stream to a console sink using writeStream.

Configure output modes and trigger intervals.

3. Pipeline Skeleton Script

Copy this starter template to write the data transformation logic:

python

# starter.py - Streaming Order Processing
from pyspark.sql import SparkSession
from pyspark.sql.functions import col

def process_stream():
    spark = SparkSession.builder.appName("StreamingLab").getOrCreate()
    # TODO: Configure streaming source, filter amounts > 5000, write stream
    pass

if __name__ == "__main__":
    process_stream()

4. Interactive Solution & Execution Output

Reveal Solution Pipeline Code

Implement the correct transformations using PySpark to achieve the aggregated values metrics output.

Pipeline Flow

Streaming Source

↓

readStream

↓

Filter amount > 5000

↓

writeStream (append mode)

↓

Console Output Sink

Step-by-Step Implementation Guide

Step 1

Define the input data schema matching the streaming source.

Step 2

Filter out rows where amount > 5000 to flag high-value, potentially fraudulent charges.

Step 3

In a real environment, you would use readStream and writeStream to run this as a continuous streaming job. Here, we run it as a batch query on the delta log to verify the logic.

Step 4

Print the flagged fraudulent records to console.

Common Mistakes

Using operations that require the full dataset (like orderBy) in streaming queries without defining window watermarks.
Forgetting to specify the checkpoint directory, causing stream processing jobs to fail on restart.

Interviewer's Expectations

Why this is asked:

Structured Streaming is the standard tool for processing high-throughput real-time data feeds.

What they look for:

Awareness of readStream, writeStream, checkpointing, and streaming aggregations restrictions.

Common mistakes:

Using non-streaming operations (like sorting global datasets) that throw errors in streaming contexts.

An excellent answer includes:

Explain streaming watermarks, stateful processing, and why checkpointing is critical for fault tolerance.

✔ What You Learned

✔Configure streaming inputs and outputs
✔Write real-time filter logic
✔Set up streaming checkpoints

Continue Learning

Previous LabIncremental ETL Pipeline Related Theory LessonIntroduction to Structured Streaming Next LabLarge-Scale Customer Analytics