JSON
Overview
Learn how to read and write semi-structured JSON files in Spark, handle nested schemas, and parse line-delimited records.
What You Will Learn
In this lesson, you will learn:
- JSON parsing: Loading line-delimited vs multi-line JSON structures.
- Nested Schema: Accessing nested JSON attributes using dot notation.
- Schema Structs: Mapping complex objects.
Detailed Concept Explanation
By default, Spark expects JSON files to be line-delimited (also called JSON Lines or NDJSON), where each line contains exactly one valid JSON object:
{"id": 1, "name": "Alice"}
{"id": 2, "name": "Bob"}
If your JSON file contains a single large JSON array or a nested structure spanning multiple lines, you must enable the multiLine option:
spark.read.option("multiLine", "true").json("file.json")
Code Examples
Input Dataset Preview
Below is the users nested JSON structure users.json:
{"id": 1, "name": "Alice", "contact": {"email": "alice@gmail.com", "phone": "123"}}
{"id": 2, "name": "Bob", "contact": {"email": "bob@gmail.com", "phone": "456"}}
Python (PySpark) Implementation
from pyspark.sql import SparkSession
from pyspark.sql.functions import col
spark = SparkSession.builder.appName("JSONRead").getOrCreate()
df = spark.read.format("json").load("users.json")
# Extract nested contact email
email_df = df.select(col("id"), col("contact.email").alias("email"))
email_df.show()
Expected Output
+---+---------------+
| id| email|
+---+---------------+
| 1|alice@gmail.com|
| 2| bob@gmail.com|
+---+---------------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("JSONScala").getOrCreate()
val df = spark.read.format("json").load("users.json")
val emails = df.select($"id", $"contact.email".as("email"))
emails.show()
Expected Output
+---+---------------+
| id| email|
+---+---------------+
| 1|alice@gmail.com|
| 2| bob@gmail.com|
+---+---------------+
SQL Implementation
SELECT id, contact.email AS email
FROM json.`users.json`;
Expected Output
Executing this SQL query returns:
| id | |
|---|---|
| 1 | alice@gmail.com |
| 2 | bob@gmail.com |
Common Mistakes
- Failing on Standard Pretty JSON: Trying to read pretty-printed multi-line JSON structures without setting the
multiLineoption. Spark will fail or read each line as a corrupted record.
Best Practices
- Define Struct Types for Nesting: For complex nested JSON, define a programmatic schema explicitly to avoid expensive pre-scan type discovery.
Interview Perspective
How do you read a nested attribute from a JSON DataFrame in Spark?
You can access nested fields directly using dot notation within your column selections (e.g. col("parent.child")). Spark automatically maps nested JSON structures into StructType columns.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.