beginner

Creating DataFrames

By DataPlayArena Technical Review BoardUpdated 2026-07-088 min readRuntime Verified

Overview

Learn the different ways to create Spark DataFrames: from raw lists, local collections, or directly from files.

What You Will Learn

In this lesson, you will learn:
  • Create from Collections: Converting local lists or tuples to DataFrames.
  • Specify Column Names: Defining column structures.
  • Create from Existing RDDs: Bridging low-level APIs to DataFrames.

Detailed Concept Explanation

You can create DataFrames using the createDataFrame method on the SparkSession. This takes a local collection (like a list of tuples in Python or Seq in Scala) and maps it to rows. Optionally, you can pass a list of column names or a StructType schema.


Code Examples

Input Dataset Preview

Below is the local collection data representing products in stock:

itemIditemprice
1Pen1.5
2Book15.0

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("CreateDF").getOrCreate()
data = [(1, "Pen", 1.5), (2, "Book", 15.0)]
df = spark.createDataFrame(data, ["itemId", "item", "price"])
df.show()

Expected Output

text
+------+----+-----+
|itemId|item|price|
+------+----+-----+
|     1| Pen|  1.5|
|     2|Book| 15.0|
+------+----+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
createDataFrame
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("CreateDFScala").getOrCreate()
import spark.implicits._

val df = Seq((1, "Pen", 1.5), (2, "Book", 15.0)).toDF("itemId", "item", "price")
df.show()

Expected Output

text
+------+----+-----+
|itemId|item|price|
+------+----+-----+
|     1| Pen|  1.5|
|     2|Book| 15.0|
+------+----+-----+

SQL Implementation

sql
-- Creating virtual tables from values
SELECT 1 AS itemId, 'Pen' AS item, 1.5 AS price
UNION ALL
SELECT 2 AS itemId, 'Book' AS item, 15.0 AS price;

Expected Output

Executing this SQL query returns:

itemIditemprice
1Pen1.5
2Book15.0

Common Mistakes

  • Mismatching Data Lengths: Passing column names list with a different length than the tuple items will throw a runtime execution error.

Best Practices

  • Explicit Types: In production, prefer defining a strict StructType schema rather than letting Spark infer types from local lists.

Interview Perspective

How do you convert a local Python collection into a Spark DataFrame?

Use spark.createDataFrame(data, schema). You pass the local python list of tuples as data, and optionally pass a list of column names or a StructType schema to define data types explicitly.


Interactive Challenges

Challenge 1: Convert List (Beginner)

What method is called on the SparkSession object to convert a local list of tuples into a DataFrame?

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.