ORC
Overview
ORC (Optimized Row Columnar) is a columnar storage format highly optimized for Apache Hive and Hadoop ecosystems.
What You Will Learn
In this lesson, you will learn:
- ORC Architecture: Columnar storage, stripes, and footer structures.
- Hive Integration: Why ORC is preferred in legacy Hadoop systems.
- Compression: Comparing ORC's ZLIB compression to Parquet's Snappy.
Detailed Concept Explanation
Similar to Parquet, ORC is a columnar binary file format. It divides files into groups of rows called Stripes, which contain column data, index details, and footer statistics.
Parquet vs ORC
- Ecosystem: Parquet is a general open-source standard designed for the entire Hadoop/Spark ecosystem. ORC was originally designed for Apache Hive and remains highly optimized for Hive queries.
- Compression: ORC commonly uses ZLIB compression by default, which yields smaller file sizes than Snappy but requires more CPU overhead to decompress.
Code Examples
Input Dataset Preview
Below is the users transactional dataset:
| userId | status |
|---|---|
| 1 | active |
| 2 | pending |
Python (PySpark) Implementation
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("ORCTest").getOrCreate()
data = [(1, "active"), (2, "pending")]
df = spark.createDataFrame(data, ["userId", "status"])
# Save DataFrame as ORC
df.write.format("orc").mode("overwrite").save("users.orc")
# Read ORC
read_df = spark.read.format("orc").load("users.orc")
read_df.show()
Expected Output
+------+-------+
|userId| status|
+------+-------+
| 1| active|
| 2|pending|
+------+-------+
Execution Plan Diagram (Python & Scala)
Scala Implementation
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("ORCScala").getOrCreate()
import spark.implicits._
val df = Seq((1, "active"), (2, "pending")).toDF("userId", "status")
df.write.format("orc").mode("overwrite").save("users.orc")
val readDF = spark.read.format("orc").load("users.orc")
readDF.show()
Expected Output
+------+-------+
|userId| status|
+------+-------+
| 1| active|
| 2|pending|
+------+-------+
SQL Implementation
-- Query an ORC directory directly
SELECT *
FROM orc.`users.orc`
WHERE status = 'active';
Expected Output
Executing this SQL query returns:
| userId | status |
|---|---|
| 1 | active |
Common Mistakes
- Applying inferSchema to ORC: Similar to Parquet, ORC files store their schema natively, so using
inferSchemais redundant.
Best Practices
- Use in Hive Environments: Use ORC when your Spark jobs read from or write to legacy Apache Hive tables to maximize partition compatibilities.
Interview Perspective
What are ORC Stripes and how do they help query optimization?
ORC files are divided into independent row groups called stripes (typically 64MB-250MB). Each stripe contains its own index and statistics (min/max values). During queries, Spark uses these stripe-level statistics to skip entire stripes if they do not match filter conditions, reducing data read from disk.
Interactive Challenges
Related Topics
DataPlayArena Technical Editorial Board
Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.
Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.