Creating RDDs

Overview

Learn the two primary ways to create RDDs: by parallelizing local memory collections or by reading external files.

What You Will Learn

Detailed Concept Explanation

You can initialize RDDs in two ways:

1. sc.parallelize(collection, partitions): Takes a local memory list or collection and distributes it across worker executors.
2. sc.textFile("path"): Reads a text file (from local disk, HDFS, or cloud storage) and creates an RDD where each record represents a line of text.

Code Examples

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CreateRDD").getOrCreate()
sc = spark.sparkContext

# Method 1: Parallelize local list
rdd1 = sc.parallelize(["London", "Paris", "Berlin"], numSlices=2)

# Method 2: Read external text file
# rdd2 = sc.textFile("cities.txt")

print("Partitions Count:", rdd1.getNumPartitions())

Expected Output

Partitions Count: 2

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CreateRDDScala").getOrCreate()
val sc = spark.sparkContext

val rdd = sc.parallelize(Seq("London", "Paris", "Berlin"), numSlices = 2)
println(s"Partitions Count: ${rdd.getNumPartitions()}")

Expected Output

Partitions Count: 2

SQL Perspective

df = rdd1.map(lambda x: (x,)).toDF(["city"])
df.createOrReplaceTempView("cities_table")
spark.sql("SELECT * FROM cities_table").show()

Common Mistakes

• Parallelizing Massive Lists: Attempting to parallelize a massive dataset (e.g. 50GB list in Python memory) using parallelize(). This will overload the driver node. Use textFile or load from databases directly.

Best Practices

• Specify numSlices: Always specify the partition count parameter (numSlices or minPartitions) explicitly to balance the work across executors.

Interview Perspective

Interactive Challenges

Related Topics

• Introduction to RDD
• RDD Transformations
• Reading Data