Defining Schemas

Overview

Learn how to define schemas programmatically using Spark's StructType and StructField classes. This improves query performance and ensures data quality.

What You Will Learn

Detailed Concept Explanation

Instead of letting Spark guess data types (via inferSchema), you can define your schema programmatically. This is called a User-Defined Schema.

Spark represents schemas using three core classes:

1. StructType: Represents the overall table structure (a collection of fields).
2. StructField: Represents a single column, specifying its name, data type, and whether it can contain null values.
3. DataType: The specific type of the column (e.g. IntegerType, StringType, DoubleType).

By passing this schema to your reader (spark.read.schema(customSchema)), you save Spark from executing a pre-scan over your data.

Code Examples

Input Dataset Preview

Below is the strict schema dataset we want to load:

Python (PySpark) Implementation

from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, StringType, DoubleType

spark = SparkSession.builder.appName("DefinedSchema").getOrCreate()

# Define schema programmatically
schema = StructType([
    StructField("id", IntegerType(), False),
    StructField("item", StringType(), True),
    StructField("price", DoubleType(), True)
])

# Load data with the schema
data = [(1, "Cup", 4.99), (2, "Plate", 9.50)]
df = spark.createDataFrame(data, schema)
df.printSchema()

Expected Output

Spark builds the schema exactly as defined:

root
 |-- id: integer (nullable = false)
 |-- item: string (nullable = true)
 |-- price: double (nullable = true)

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.types._

val spark = SparkSession.builder().appName("SchemaScala").getOrCreate()

val schema = StructType(Array(
  StructField("id", IntegerType, nullable = false),
  StructField("item", StringType, nullable = true),
  StructField("price", DoubleType, nullable = true)
))

val df = spark.createDataFrame(
  spark.sparkContext.parallelize(Seq(Row(1, "Cup", 4.99), Row(2, "Plate", 9.50))),
  schema
)
df.printSchema()

Expected Output

root
 |-- id: integer (nullable = false)
 |-- item: string (nullable = true)
 |-- price: double (nullable = true)

SQL Implementation

-- Defining a schema inline inside SQL DDL
CREATE TABLE products_catalog (
    id INT,
    item STRING,
    price DOUBLE
) USING parquet;

Expected Output

Registers an SQL catalog table with specified schema types.

Common Mistakes

• Data Type Mismatches: If you pass a programmatic schema specifying a column is an IntegerType but the raw file contains string text (e.g. "N/A"), Spark may fill that column with nulls.

Best Practices

• Use Nullable Flag Safely: Mark the nullable flag inside StructField as true unless you are absolutely sure the source files will never contain missing values.

Interview Perspective

Interactive Challenges

Related Topics

• Schema Inference
• Reading Data
• Creating DataFrames