History of Apache Spark
Overview
Understanding the history of Apache Spark helps clarify why its architecture was designed the way it is. Spark was created as a research project to solve the performance and usability limitations of Hadoop MapReduce.
What You Will Learn
In this lesson, you will learn:
- Spark's Origin: Where Spark was created and the research goals behind it.
- MapReduce Limitations: The specific problems Spark's creators set out to fix.
- Apache Foundation: When Spark became an open-source standard.
- Databricks Evolution: The creation of Databricks and its role in modern Spark development.
Detailed Concept Explanation
The UC Berkeley AMPLab Beginnings
Apache Spark was started in 2009 as a research project at UC Berkeley's AMPLab by Matei Zaharia. At the time, Hadoop MapReduce was the dominant big data engine.
Zaharia and his team observed two major problems with MapReduce:
- Inefficiency for Iterative Workloads: Algorithms that read the same dataset repeatedly (such as machine learning training or graph analysis) were extremely slow. This was because MapReduce wrote all intermediate data to physical storage disks between each step.
- Complex and Fragmented APIs: Developers had to write custom Map and Reduce functions for every simple query, and separate engines were needed for SQL, streaming, and batch computations.
The Open Source Journey
- 2010: Spark was open-sourced under a BSD license.
- 2013: The project was donated to the Apache Software Foundation and re-licensed under Apache 2.0.
- 2014: Spark became an Apache Top-Level Project. In November 2014, Spark set a world record in large-scale sorting, sorting 100TB of data 3 times faster than Hadoop MapReduce using only 10% of the machines.
The Rise of Databricks
In 2013, the creators of Spark founded Databricks. Databricks provided a managed cloud platform and interactive notebook interface, making it easy for organizations to deploy and scale Spark clusters. Databricks continues to be the primary corporate contributor to the open-source Spark codebase, driving major releases like Spark 2.0 (introducing structured APIs) and Spark 3.0 (introducing Adaptive Query Execution).
Common Mistakes
- Assuming Spark Replaced Hadoop: Spark did not replace Hadoop entirely. Spark replaces the compute layer (MapReduce). It still frequently integrates with Hadoop's storage layer (HDFS) and resource manager (YARN).
Best Practices
- Understand Version Support: When building production systems, stick to stable Apache Spark 3.x+ releases, which include advanced Adaptive Query Execution (AQE) features.
Interview Perspective
Why was Spark created if Hadoop MapReduce was already widely adopted?
Hadoop MapReduce was revolutionary for distributed storage and processing but was limited by its disk-based compute model. Every Map and Reduce step required writing to local disk, making multi-stage and iterative algorithms extremely slow. Spark was created to introduce a unified, in-memory computing model that caches intermediate steps in RAM, eliminating the disk I/O bottleneck.