Schema Inference
Overview
Learn how Spark automatically guesses the data types of columns when reading text-based files, and understand the trade-offs of using schema inference.
What You Will Learn
In this lesson, you will learn:
- Inference Definition: How Spark determines schema types automatically.
- Trade-Offs: The performance and cost implications of reading files twice.
- Type Guessing: How header scanning resolves integers vs string types.
Detailed Concept Explanation
When reading structured text files (such as CSV or JSON) without specifying a schema, Spark can be configured to infer it:
spark.read.option("inferSchema", "true")
To do this, Spark performs an extra pass over the dataset, reading the records to determine the most compatible data type for each column (e.g. integer vs float vs string).
While convenient, schema inference has two drawbacks:
- Performance cost: For huge files, reading the dataset twice creates a massive disk I/O bottleneck.
- Inaccuracy: A column containing mixed values might be incorrectly cast, leading to query failures.
Code Examples
Input Dataset Preview
Assume a CSV file prices.csv contains values:
id,item,cost
1,Apple,0.99
2,Orange,1.25
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("SchemaInference").getOrCreate()
df = spark.read.format("csv") \
.option("header", "true") \
.option("inferSchema", "true") \
.load("prices.csv")
# Print the inferred schema types
df.printSchema()
Expected Output
Spark prints the types it inferred by scanning the costs column:
root
|-- id: integer (nullable = true)
|-- item: string (nullable = true)
|-- cost: double (nullable = true)
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("InferenceScala").getOrCreate()
val df = spark.read.format("csv")
.option("header", "true")
.option("inferSchema", "true")
.load("prices.csv")
df.printSchema()
Expected Output
root
|-- id: integer (nullable = true)
|-- item: string (nullable = true)
|-- cost: double (nullable = true)
SQL Implementation
-- SQL direct query on inferred CSV schema
SELECT *
FROM csv.`prices.csv`
WHERE cost > 1.0;
Expected Output
Executing this SQL query returns:
| id | item | cost |
|---|---|---|
| 2 | Orange | 1.25 |
Common Mistakes
- Using Inference in Production: Enabling
inferSchemaon multi-terabyte datasets can cause jobs to stall because Spark has to read the entire dataset over the network just to determine the schema before processing it.
Best Practices
- Define Explicit Schemas: In production, always define schemas programmatically using
StructTypeto skip the schema discovery scan.
Interview Perspective
What are the performance implications of setting inferSchema to true in Spark?
Setting inferSchema to true forces Spark to execute a pre-job scan over the dataset to evaluate column values. This requires reading the file twice, creating a major I/O bottleneck on large datasets. In production, schemas should be defined explicitly.