Last-Minute Spark Revision Sheet

Review core architectures, operations, shuffles, optimizations, and syntax cheat sheets within 15 minutes before your interview.

15-Min Quick Read

1. Spark Fundamentals (Quick Revision)

Apache Spark is an in-memory, distributed general-purpose data engine. It runs operations in worker memory nodes instead of writing files to local disk spindles between phases, making it up to 100 times faster than Hadoop MapReduce.

Concept	Revision Summary
Driver Node	The main orchestrator process that runs the main() method, creates SparkSession, plans logical operations, schedules tasks, and coordinates worker executors.
Executors	Worker processes running on worker machines. They store dataset partitions in local RAM memory and run physical tasks.
Partitions	The logical chunks that a large dataset is divided into. Each executor thread processes one partition at a time.

2. Spark Architecture Map

This map illustrates how the Driver coordinates with the Cluster Manager (like YARN or Kubernetes) to allocate worker Executors across multiple physical server machines.

[ Driver Process ]

SparkSession / DAG / TaskScheduler

↔

[ Cluster Manager ]

YARN / K8s / Standalone

↔

Exec Executor 1 (Tasks)

Exec Executor 2 (Tasks)

3. Spark Execution Flow

When you run a Spark job, it travels through several stages of transformation: from parsing raw code into logical graphs to executing physical worker tasks.

1. Code Submission (spark-submit launches Driver)

↓

2. Logical Plan DAG Generation ( Catalyst optimization stages )

↓

3. Stage Partitioning ( Wide boundaries splits DAG into stages )

↓

4. Task Distribution ( Tasks dispatched to workers for execution )

4. DataFrame Syntax Cheatsheet

python

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Create Spark Session
spark = SparkSession.builder.appName("ETL").getOrCreate()

# Read CSV data with options
df = spark.read.csv("data.csv", header=True, inferSchema=True)

# Select, filter, and calculate bonus using parentheses chain
result = (df.select("name", "salary")
            .filter(col("salary") > 80000)
            .withColumn("bonus", col("salary") * 0.1))

# Write output partitioned by department
result.write.mode("overwrite").partitionBy("department").parquet("output.parquet")

5. RDD Syntax Cheatsheet

python

# Read as Resilient Distributed Dataset (RDD)
rdd = spark.sparkContext.textFile("logs.txt")

# Chain map operations to perform word count
words = (rdd.flatMap(lambda line: line.split(" "))
            .map(lambda word: (word, 1))
            .reduceByKey(lambda a, b: a + b))

# Collect results to a local Python list
output = words.collect()

6. Spark SQL Cheatsheet

python

# Create a Temporary Table View
df.createOrReplaceTempView("employees")

# Execute standard SQL query
sql_df = spark.sql("SELECT department, AVG(salary) as avg_sal FROM employees GROUP BY department HAVING avg_sal > 90000")
sql_df.show()

7. Window Functions Cheatsheet

python

from pyspark.sql.window import Window
from pyspark.sql.functions import col, dense_rank, lag

# Define partition boundaries and ordering
windowSpec = Window.partitionBy("department").orderBy(col("salary").desc())

# Apply rank and lookback value over window specifications
ranked_df = (df.withColumn("rank", dense_rank().over(windowSpec))
               .withColumn("prev_sal", lag("salary", 1).over(windowSpec)))

8 & 9. Most Used API Methods

Understand the difference between lazy transformations (which construct logic) and actions (which trigger data execution).

Transformations (Lazy)

select()	Project column layouts.
filter()	Filter rows on conditions.
groupBy()	Group records (triggers shuffles).
join()	Join relational tables.

Actions (Triggers Job)

show()	Print lines to console.
count()	Return integer total.
collect()	Send all rows to driver RAM.
write()	Trigger folder output write steps.

10. Read & Write APIs

python

# Read APIs
spark.read.parquet("folder/*.parquet")
spark.read.json("logs.json")

# Write APIs with Modes
df.write.mode("overwrite").csv("target_dir")
df.write.mode("append").saveAsTable("hive_db.table_name")

11. Join Types Comparison

Choosing the right join type determines the amount of data transferred across the cluster network.

Join Type	Behavior	Shuffle Tradeoff
Inner	Keeps matched rows in both datasets.	Requires sorting/shuffling unless pre-bucketed.
Left / Right	Preserves rows on outer side, fills NULLs on unmatched inner.	SortMergeJoin triggers standard network shuffles.
Left Anti	Keeps rows on left table that have NO match on right.	Optimized way to isolate delta changes.
Broadcast	Replicates smaller table to worker memory blocks.	Zero Network Shuffling on the big table. Highly performant.

12. Performance Tuning Checklist

These performance guidelines are crucial to understand for writing efficient jobs and answering design questions.

Broadcast Small Lookups: Wrap lookup tables under 10MB in broadcast() joins to bypass network shuffling entirely.

Deduplicate Prior to Joins: Drop duplicates and filter data early to reduce record payloads before joins.

Mitigate Key Skews: Salt hot keys with random integers to distribute partitions evenly.

Leverage Caching: Cache intermediate DataFrames that are reused multiple times in the pipeline.

Additional Production Tuning Items:

Kryo Serialization: Configure spark.serializer to KryoSerializer. It is up to 10x faster and more compact than standard Java serialization.

Dynamic Partition Pruning (DPP): Skips scanning irrelevant partitions dynamically during joins (critical in star-schema lookups).

Vectorized Parquet Reader: Enables batch decompression and decoding of columns directly into memory blocks.

Garbage Collection Tuning (G1GC): Configure worker JVMs to run the G1GC garbage collector. This prevents long, blocking GC pauses that can cause executor heartbeats to timeout.

13. Shuffle Quick Notes

Shuffle is the process of redistributing data across executors. It is triggered by wide operations (e.g. groupBy, join, distinct, repartition).

Key Interview Notes:

Shuffles involve writing partition blocks to local disk and copy operations over the network (highly I/O expensive).
Always minimize shuffles by aggregating early, using map-side broadcast joins, or bucketing tables.

14. Narrow vs Wide Transformations

Transformations are divided into narrow (independent) and wide (dependent) categories based on their shuffle dependencies.

Metric	Narrow Transformation	Wide Transformation
Definition	Each input partition maps to exactly one output partition.	Multiple input partitions contribute to single output partitions.
Network Shuffle	No Shuffle required. Tasks run locally in parallel.	Triggers network shuffle. High latency operations.
Examples	select(), filter(), map(), flatMap(), union()	groupBy(), join(), distinct(), repartition(), coalesce()

15. Cache vs Persist

Use caching to save intermediate calculations if a DataFrame is referenced multiple times in a session workflow.

Feature	cache()	persist()
Storage Level	Hardcoded to memory storage (MEMORY_AND_DISK).	Accepts custom StorageLevel objects.
Options	None. Simple default shortcut.	MEMORY_ONLY, DISK_ONLY, MEMORY_ONLY_SER, etc.

16. Repartition vs Coalesce

Repartitioning reorganizes data entirely, whereas Coalesce merges partitions without shuffling.

Aspect	repartition()	coalesce()
Operation Size	Can increase or decrease partition count.	Can only decrease partition count.
Shuffle	Full Shuffle. Distributes data evenly across network.	No Shuffle. Merges local partition blocks.
Best For	Balancing skewed partitions before joins.	Shrinking partition sizes right before writing files.

17. Broadcast Join

python

from pyspark.sql.functions import broadcast

# Join small table lookup (under 10MB) to large table
# Replicates users lookup to executor memory, bypassing shuffles
optimized_df = transactions.join(broadcast(users), on="user_id", how="inner")

18. Lazy Evaluation

Spark transformations are evaluated lazily. This means Spark registers operations in a logical DAG without executing them immediately. Execution is deferred until an Action (like show, count, collect) is called.

Why it matters:

This allows the Catalyst Optimizer to inspect the complete lineage graph, combining filter operations or pushing down projections to minimize physical data loading.

19. Catalyst Optimizer

This plan flow shows how Spark translates code instructions into an optimized physical execution schedule.

1. Unresolved Logical Plan (Syntax checks)

↓

2. Resolved Logical Plan (Catalog schema matches)

↓

3. Optimized Logical Plan (Filter pushdowns, constant folds)

↓

4. Physical Plans Selection (Pick cheapest cost plan)

20. AQE (Adaptive Query Execution)

Enabled by default in Spark 3, AQE re-optimizes logical execution plans dynamically based on runtime metrics collected during shuffle stages.

Coalesce PartitionsShrinks empty post-shuffle partitions automatically to prevent task creation overhead.

Optimize Join TypesConverts SortMergeJoin dynamically into Broadcast Hash Joins if runtime partition size is small.

Deduplicate SkewsSplits skewed partitions into smaller sub-partitions to distribute work across worker cores.

21. Common Spark Mistakes

Calling collect() on large datasets:This pulls all rows into the driver node's RAM, causing out-of-memory crashes. Use count(), show(), or write() instead.

Using Python UDFs: Standard Python UDFs execute outside the JVM, forcing expensive serialization cycles. Always prefer built-in functions.

Ignoring partition counts: Too few partitions create giant chunks (causing memory overhead), while too many partitions overload the coordinator with thread-scheduling tasks.

22. Interview Red Flags (Candidate Don'ts)

Red Flag: Recommending coalesce() to increase partition count. *coalesce does not trigger shuffles, meaning it cannot increase partition count. Always use repartition() to increase.*
Red Flag: Confusing map() with mapPartitions(). *mapPartitions initializes external connections once per partition block instead of once per row, saving substantial connection setup overhead.*
Red Flag: Suggesting scaling up hardware specs as the first solution for data skew issues without checking the execution plan first.

23. One-Page Final Revision (1-Min Read)

✔ Spark Context runs inside the Driver. Tasks execute on Executors.

✔ Transformations register in logical lineages (lazy). Actions trigger evaluation.

✔ Wide transformations require network shuffles. Narrow transformations execute locally.

✔ Resolve skew joins using salting or broadcast hash joins.

✔ Coalesce decreases partition counts without triggering full network shuffles.

24. Scenario Interview Prompts Simulator

Interactive

Select a common data engineering interview scenario prompt below to see how a Senior Data Engineer would structure their explanation:

Interviewer Prompt:

"Our pipeline runs fine on Sunday but crashes with an OOM error on Monday morning. What do you suspect?"

Underlying Issue:

Weekly data volume spikes on Monday mornings due to weekend accumulation, triggering data skew on popular keys.

Your Response Checklist:

Check for data volume skew using Spark Web UI task execution timeline.
Check partition sizes. Monday data might need repartitioning or key-salting.
Ensure Adaptive Query Execution (AQE) is enabled to handle runtime partition coalescing.
Check for memory-heavy operations like collect() or large broadcast joins.