Creating DataFrames
Overview
Learn the different ways to create Spark DataFrames: from raw lists, local collections, or directly from files.
What You Will Learn
In this lesson, you will learn:
- Create from Collections: Converting local lists or tuples to DataFrames.
- Specify Column Names: Defining column structures.
- Create from Existing RDDs: Bridging low-level APIs to DataFrames.
Detailed Concept Explanation
You can create DataFrames using the createDataFrame method on the SparkSession. This takes a local collection (like a list of tuples in Python or Seq in Scala) and maps it to rows. Optionally, you can pass a list of column names or a StructType schema.
Code Examples
Input Dataset Preview
Below is the local collection data representing products in stock:
| itemId | item | price |
|---|---|---|
| 1 | Pen | 1.5 |
| 2 | Book | 15.0 |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("CreateDF").getOrCreate()
data = [(1, "Pen", 1.5), (2, "Book", 15.0)]
df = spark.createDataFrame(data, ["itemId", "item", "price"])
df.show()
Expected Output
+------+----+-----+
|itemId|item|price|
+------+----+-----+
| 1| Pen| 1.5|
| 2|Book| 15.0|
+------+----+-----+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("CreateDFScala").getOrCreate()
import spark.implicits._
val df = Seq((1, "Pen", 1.5), (2, "Book", 15.0)).toDF("itemId", "item", "price")
df.show()
Expected Output
+------+----+-----+
|itemId|item|price|
+------+----+-----+
| 1| Pen| 1.5|
| 2|Book| 15.0|
+------+----+-----+
SQL Implementation
-- Creating virtual tables from values
SELECT 1 AS itemId, 'Pen' AS item, 1.5 AS price
UNION ALL
SELECT 2 AS itemId, 'Book' AS item, 15.0 AS price;
Expected Output
Executing this SQL query returns:
| itemId | item | price |
|---|---|---|
| 1 | Pen | 1.5 |
| 2 | Book | 15.0 |
Common Mistakes
- Mismatching Data Lengths: Passing column names list with a different length than the tuple items will throw a runtime execution error.
Best Practices
- Explicit Types: In production, prefer defining a strict
StructTypeschema rather than letting Spark infer types from local lists.
Interview Perspective
How do you convert a local Python collection into a Spark DataFrame?
Use spark.createDataFrame(data, schema). You pass the local python list of tuples as data, and optionally pass a list of column names or a StructType schema to define data types explicitly.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.