Why Apache Spark?

Overview

As data volumes grow exponentially, legacy data processing systems like traditional relational databases and Hadoop MapReduce encounter severe performance bottlenecks. Apache Spark was designed specifically to overcome these bottlenecks, offering unmatched processing speed, developer-friendly APIs, and a unified execution engine for diverse data workloads.

What You Will Learn

Detailed Concept Explanation

The Big Data Evolution: Overcoming MapReduce Bottlenecks

To understand why Spark is so dominant today, we must first look at what came before it: Hadoop MapReduce.

While MapReduce revolutionized distributed storage and batch computation, it suffered from a major architectural limitation: excessive disk I/O. MapReduce jobs are divided into rigid, single-step Map and Reduce phases. If you need to perform multi-stage iterative algorithms (such as machine learning training or multi-step ETL pipelines):

1. The Map phase reads data from HDFS (disk).
2. The Map phase writes intermediate partitions back to local storage disks.
3. The Reduce phase reads those intermediate partitions over the network.
4. The Reduce phase writes final outputs back to HDFS (disk).
5. The next Map job begins by reading from disk again.

This constant writing to and reading from disk spindles creates a massive network and disk I/O bottleneck.

How Spark Solves the Disk Bottleneck

Apache Spark introduces two primary concepts that eliminate this disk dependency:

1. In-Memory Computing: Spark allows datasets to be cached in the RAM memory of executor worker nodes. Subsequent read actions can query the RAM memory directly, bypassing disk reads entirely. This makes Spark up to 100x faster than MapReduce for iterative workloads.
2. Directed Acyclic Graph (DAG) Engine: Instead of forcing jobs into strict map-then-reduce stages, Spark's Catalyst Optimizer compiles your DataFrame transformations into a complex dependency pipeline (a DAG). Spark can combine operations (such as chaining multiple filters and column mappings) and execute them in a single stage, minimizing network shuffles.

Architecture Diagram

Code Examples

Input Dataset Preview

Before executing the query implementations below, let's preview the input dataset schema and rows:

Python (PySpark) Implementation

Using PySpark's declarative DataFrame API, we can chain operations naturally:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# Initialize Spark Session
spark = SparkSession.builder.appName("WhySparkExample").getOrCreate()

# Create a mock DataFrame
data = [("Alice", "Engineering", 120000), 
        ("Bob", "Marketing", 90000), 
        ("Charlie", "Engineering", 110000)]
df = spark.createDataFrame(data, ["name", "dept", "salary"])

# Filter for salaries greater than 100,000 and select columns
high_earners = df.filter(col("salary") > 100000).select("name", "salary")
high_earners.show()

Expected Output

Spark executes the filters in memory and outputs the following dataset partition format:

+-------+------+
|   name|salary|
+-------+------+
|  Alice|120000|
|Charlie|110000|
+-------+------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

Because Spark is written in Scala, the Scala API is highly optimized, typesafe, and concise:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("WhySparkScala").getOrCreate()
import spark.implicits._

val df = Seq(
  ("Alice", "Engineering", 120000),
  ("Bob", "Marketing", 90000),
  ("Charlie", "Engineering", 110000)
).toDF("name", "dept", "salary")

val highEarners = df.filter($"salary" > 100000).select("name", "salary")
highEarners.show()

Expected Output

Scala executes the operations on the JVM and yields:

+-------+------+
|   name|salary|
+-------+------+
|  Alice|120000|
|Charlie|110000|
+-------+------+

SQL Implementation

Spark SQL allows developers to query DataFrames using ANSI-compliant SQL expressions, leveraging the Catalyst Optimizer under the hood:

-- Registering the DataFrame as a virtual table
-- df.createOrReplaceTempView("employees")

SELECT name, salary 
FROM employees 
WHERE salary > 100000;

Expected Output

Executing this SQL query returns the filtered result:

Common Mistakes

• Unintentional Action Triggers: Because Spark is lazily evaluated, calling actions like .collect() or .count() inside loops triggers a full job execution, nullifying the DAG optimizer benefits.
• Out of Memory (OOM) Errors: Bringing massive datasets back to the driver node using .collect() instead of writing to a storage sink will crash the Spark driver process.

Performance Notes

• Shuffles are Expensive: Any transformation requiring data redistribution across keys (such as groupBy() or join()) forces a physical shuffle. Always filter datasets as early as possible to minimize the volume of data transferred over the network.
• Use Coalesce Over Repartition: If you want to decrease the number of partitions (e.g. before saving a file), use .coalesce(), which avoids a full shuffle, unlike .repartition().

Best Practices

• Filter Early: Apply .filter() or .where() clauses as early as possible in your transformation chains to minimize intermediate data size before a shuffle boundary.
• Do Not Over-Cache: While caching speeds up iterative jobs, storing unnecessary DataFrames in memory consumes executor heap space, which can trigger garbage collection (GC) pauses or OOM issues.
• Leverage Built-in Functions: Avoid writing user-defined functions (UDFs) in Python. PySpark UDFs run outside the JVM, forcing slow data serialization between Python and the JVM. Use built-in pyspark.sql.functions instead.

Interview Perspective

Interactive Challenges

df1 = df.filter(col("age") > 21)
df2 = df1.withColumn("salary_taxed", col("salary") * 0.8)
df3 = df2.groupBy("department").avg("salary_taxed")

Related Topics

• Spark Core Architecture
• Web UI & Monitoring
• Installation & Local Setup