JSON

Overview

Learn how to read and write semi-structured JSON files in Spark, handle nested schemas, and parse line-delimited records.

What You Will Learn

Detailed Concept Explanation

By default, Spark expects JSON files to be line-delimited (also called JSON Lines or NDJSON), where each line contains exactly one valid JSON object: {"id": 1, "name": "Alice"} {"id": 2, "name": "Bob"}

If your JSON file contains a single large JSON array or a nested structure spanning multiple lines, you must enable the multiLine option: spark.read.option("multiLine", "true").json("file.json")

Code Examples

Input Dataset Preview

Below is the users nested JSON structure users.json:

{"id": 1, "name": "Alice", "contact": {"email": "alice@gmail.com", "phone": "123"}}
{"id": 2, "name": "Bob", "contact": {"email": "bob@gmail.com", "phone": "456"}}

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.functions import col

spark = SparkSession.builder.appName("JSONRead").getOrCreate()
df = spark.read.format("json").load("users.json")

# Extract nested contact email
email_df = df.select(col("id"), col("contact.email").alias("email"))
email_df.show()

Expected Output

+---+---------------+
| id|          email|
+---+---------------+
|  1|alice@gmail.com|
|  2|  bob@gmail.com|
+---+---------------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("JSONScala").getOrCreate()
val df = spark.read.format("json").load("users.json")

val emails = df.select($"id", $"contact.email".as("email"))
emails.show()

Expected Output

+---+---------------+
| id|          email|
+---+---------------+
|  1|alice@gmail.com|
|  2|  bob@gmail.com|
+---+---------------+

SQL Implementation

SELECT id, contact.email AS email 
FROM json.`users.json`;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Failing on Standard Pretty JSON: Trying to read pretty-printed multi-line JSON structures without setting the multiLine option. Spark will fail or read each line as a corrupted record.

Best Practices

• Define Struct Types for Nesting: For complex nested JSON, define a programmatic schema explicitly to avoid expensive pre-scan type discovery.

Interview Perspective

Interactive Challenges

Related Topics

• CSV
• Reading Data
• Parquet