Running Your First Spark Application

Overview

Now that your local environment is configured, let's write, build, and execute your first Apache Spark application using Python, Scala, and SQL.

What You Will Learn

Detailed Concept Explanation

Building the Data Pipeline

Every Spark application follows the same three basic steps:

1. Initialize: Create a SparkSession (the entry point to Spark).
2. Transform: Define operations on your data (like filtering, sorting, or aggregating).
3. Action: Trigger execution and display or write the results (e.g. using .show()).

Code Examples

Input Dataset Preview

Before executing the code examples below, let's preview the input dataset schema and rows:

Python (PySpark) Implementation

Save the code below as app.py:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

# 1. Initialize SparkSession
spark = SparkSession.builder.appName("FirstApp").getOrCreate()

# 2. Load mock dataset
data = [("Laptop", "Electronics", 1200),
        ("Phone", "Electronics", 800),
        ("Book", "Office", 20)]
df = spark.createDataFrame(data, ["product", "category", "price"])

# 3. Filter for products costing more than 500
expensive_products = df.filter(col("price") > 500).select("product", "price")

# 4. Trigger action to show results
expensive_products.show()

# 5. Stop session
spark.stop()

Expected Output

Spark executes the query in memory and prints:

+-------+-----+
|product|price|
+-------+-----+
| Laptop| 1200|
|  Phone|  800|
+-------+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

Here is the same query written in Scala:

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("FirstAppScala").getOrCreate()
import spark.implicits._

val df = Seq(
  ("Laptop", "Electronics", 1200),
  ("Phone", "Electronics", 800),
  ("Book", "Office", 20)
).toDF("product", "category", "price")

val expensive = df.filter($"price" > 500).select("product", "price")
expensive.show()

spark.stop()

Expected Output

Scala executes the operations on the JVM and yields:

+-------+-----+
|product|price|
+-------+-----+
| Laptop| 1200|
|  Phone|  800|
+-------+-----+

SQL Implementation

Spark SQL allows you to write standard SQL queries on your DataFrames:

-- Register table
-- df.createOrReplaceTempView("products")

SELECT product, price 
FROM products 
WHERE price > 500;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Forgetting to Stop SparkSession: Keeping sessions open prevents executors from shutting down, locking cluster memory.

Best Practices

• Submit locally: Test your application on your computer using local execution before submitting it to YARN or Kubernetes: spark-submit --master local[*] app.py

Interview Perspective

Interactive Challenges

Related Topics

• Installation & Local Setup
• Web UI & Monitoring
• Introduction to Apache Spark