Creating DataFrames

Overview

Learn the different ways to create Spark DataFrames: from raw lists, local collections, or directly from files.

What You Will Learn

Detailed Concept Explanation

You can create DataFrames using the createDataFrame method on the SparkSession. This takes a local collection (like a list of tuples in Python or Seq in Scala) and maps it to rows. Optionally, you can pass a list of column names or a StructType schema.

Code Examples

Input Dataset Preview

Below is the local collection data representing products in stock:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CreateDF").getOrCreate()
data = [(1, "Pen", 1.5), (2, "Book", 15.0)]
df = spark.createDataFrame(data, ["itemId", "item", "price"])
df.show()

Expected Output

+------+----+-----+
|itemId|item|price|
+------+----+-----+
|     1| Pen|  1.5|
|     2|Book| 15.0|
+------+----+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CreateDFScala").getOrCreate()
import spark.implicits._

val df = Seq((1, "Pen", 1.5), (2, "Book", 15.0)).toDF("itemId", "item", "price")
df.show()

Expected Output

+------+----+-----+
|itemId|item|price|
+------+----+-----+
|     1| Pen|  1.5|
|     2|Book| 15.0|
+------+----+-----+

SQL Implementation

-- Creating virtual tables from values
SELECT 1 AS itemId, 'Pen' AS item, 1.5 AS price
UNION ALL
SELECT 2 AS itemId, 'Book' AS item, 15.0 AS price;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Mismatching Data Lengths: Passing column names list with a different length than the tuple items will throw a runtime execution error.

Best Practices

• Explicit Types: In production, prefer defining a strict StructType schema rather than letting Spark infer types from local lists.

Interview Perspective

Interactive Challenges

Related Topics

• Introduction to DataFrames
• Reading Data
• Defining Schemas