ORC

Overview

ORC (Optimized Row Columnar) is a columnar storage format highly optimized for Apache Hive and Hadoop ecosystems.

What You Will Learn

Detailed Concept Explanation

Similar to Parquet, ORC is a columnar binary file format. It divides files into groups of rows called Stripes, which contain column data, index details, and footer statistics.

Parquet vs ORC

• Ecosystem: Parquet is a general open-source standard designed for the entire Hadoop/Spark ecosystem. ORC was originally designed for Apache Hive and remains highly optimized for Hive queries.
• Compression: ORC commonly uses ZLIB compression by default, which yields smaller file sizes than Snappy but requires more CPU overhead to decompress.

Code Examples

Input Dataset Preview

Below is the users transactional dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("ORCTest").getOrCreate()
data = [(1, "active"), (2, "pending")]
df = spark.createDataFrame(data, ["userId", "status"])

# Save DataFrame as ORC
df.write.format("orc").mode("overwrite").save("users.orc")

# Read ORC
read_df = spark.read.format("orc").load("users.orc")
read_df.show()

Expected Output

+------+-------+
|userId| status|
+------+-------+
|     1| active|
|     2|pending|
+------+-------+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("ORCScala").getOrCreate()
import spark.implicits._

val df = Seq((1, "active"), (2, "pending")).toDF("userId", "status")
df.write.format("orc").mode("overwrite").save("users.orc")

val readDF = spark.read.format("orc").load("users.orc")
readDF.show()

Expected Output

+------+-------+
|userId| status|
+------+-------+
|     1| active|
|     2|pending|
+------+-------+

SQL Implementation

-- Query an ORC directory directly
SELECT * 
FROM orc.`users.orc`
WHERE status = 'active';

Expected Output

Executing this SQL query returns:

Common Mistakes

• Applying inferSchema to ORC: Similar to Parquet, ORC files store their schema natively, so using inferSchema is redundant.

Best Practices

• Use in Hive Environments: Use ORC when your Spark jobs read from or write to legacy Apache Hive tables to maximize partition compatibilities.

Interview Perspective

Interactive Challenges

Related Topics

• Parquet
• Avro
• Reading Data