Interview Revision Guide Reference

Quick revision overview of Apache Spark internals, configurations, and core features.

Spark Architecture & Coordinator Role

Spark operates in a Master/Worker model. The Driver coordinates execution while Workers host Executors running tasks.

Component	Responsibilities	Execution Lifecycle
Driver Node	Creates SparkSession, builds logical DAG, compiles stages, schedules tasks.	Single master process per app.
Cluster Manager	Allocates resource container blocks (YARN, Kubernetes, Standalone).	Manages clusters globally.
Executors	Stores partition blocks in RAM memory, runs physical compute tasks.	Launches on worker hosts.

Job -> Stage -> Task Hierarchy

How Spark translates user actions into physical distributed tasks.

Action: Triggers a new Spark Job execution (e.g. count, show, write).
Stage: Job is divided into Stages at Shuffle boundaries (wide transformations).
Task: Smallest unit of work, running inside an executor thread on one partition block.

Catalyst Query Optimizer Workflow

Catalyst compiles DataFrame query expressions into highly optimized physical execution instructions.

Optimization Phase	Input	Core Output
1. Analysis	Unresolved Logical Plan	Resolved Logical Plan (Schema Catalog mapped)
2. Logical Optimization	Resolved Logical Plan	Optimized Logical Plan (Filter pushdowns, projections limit)
3. Physical Planning	Optimized Logical Plan	CHEAPEST physical plans selected for executor JVMs

Narrow vs Wide Transformations

The primary divider of latency in Spark workloads.

Property	Narrow Transformation	Wide Transformation
Definition	One input partition maps to exactly one output partition.	Multiple input partitions contribute to output partitions.
Network Shuffle	No shuffle required. Tasks run locally in parallel.	Forced network shuffle. Writes blocks to local disks.
Examples	select(), filter(), map(), union()	groupBy(), join(), distinct(), repartition()

Cache vs Persist Storage Levels

Saving intermediate DataFrame structures in RAM.

Method	Default Storage Level	Customization Scope
cache()	MEMORY_AND_DISK	No configuration. Hardcoded storage helper.
persist()	MEMORY_AND_DISK	Accepts Custom StorageLevel objects (MEMORY_ONLY, DISK_ONLY, etc.)

Repartition vs Coalesce

Modifying partition sizes in a running application.

Aspect	repartition()	coalesce()
Size Change	Can increase or decrease partition numbers.	Can only decrease partition numbers.
Shuffle Stage	Forced full network shuffle.	No shuffle stage created. Merges local blocks.
Best Use Case	Balancing skewed partition sizes prior to joins.	Shrinking output files count right before writes.

Adaptive Query Execution (AQE)

Dynamic optimization features enabled in Spark 3+.

Dynamic Partition Coalescing: Shrinks empty post-shuffle partitions automatically to save task overhead.
Dynamic Join Switch: Converts SortMergeJoin dynamically into Broadcast Hash Joins if runtime partition is small.
Dynamic Skew Join Tuning: Detects skewed keys and splits partitions to prevent straggler tasks.