Apache Iceberg

Overview

Apache Iceberg is an open-source, high-performance table format designed for massive analytic datasets, providing hidden partitioning and schema evolution.

What You Will Learn

Detailed Concept Explanation

Apache Iceberg is an open table format designed for huge datasets. Unlike directory-based partitioning schemes (where queries scan files based on directory paths), Iceberg tracks the dataset files using a tree of metadata files (manifests).

Key Features

• Hidden Partitioning: Iceberg manages partitioning automatically. You don't need to specify partition columns in your queries; Iceberg handles the mappings under the hood.
• Schema Evolution: You can rename, add, or drop columns in an Iceberg table without rewriting the underlying files.
• Time Travel: Supports querying historical snapshots of your table.

Code Examples

Input Dataset Preview

Below is the users events dataset:

Python (PySpark) Implementation

from pyspark.sql import SparkSession

# Configure Iceberg catalog
spark = SparkSession.builder \
    .appName("IcebergTest") \
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions") \
    .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog") \
    .config("spark.sql.catalog.local.type", "hadoop") \
    .config("spark.sql.catalog.local.warehouse", "warehouse") \
    .getOrCreate()

data = [(1, "click")]
df = spark.createDataFrame(data, ["id", "event"])

# Create Iceberg table
df.writeTo("local.db.events").create()

# Read Iceberg table
ice_df = spark.read.table("local.db.events")
ice_df.show()

Expected Output

+---+-----+
| id|event|
+---+-----+
|  1|click|
+---+-----+

Execution Plan Diagram (Python & Scala)

Scala Implementation

import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("IcebergScala")
  .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
  .config("spark.sql.catalog.local", "org.apache.iceberg.spark.SparkCatalog")
  .config("spark.sql.catalog.local.type", "hadoop")
  .config("spark.sql.catalog.local.warehouse", "warehouse")
  .getOrCreate()

import spark.implicits._
val df = Seq((1, "click")).toDF("id", "event")

df.writeTo("local.db.events").create()

val readDF = spark.read.table("local.db.events")
readDF.show()

Expected Output

+---+-----+
| id|event|
+---+-----+
|  1|click|
+---+-----+

SQL Implementation

-- Query an Iceberg table snapshot directly in SQL
SELECT * 
FROM local.db.events;

Expected Output

Executing this SQL query returns:

Common Mistakes

• Confusing Directory Path with Table Name: Attempting to read Iceberg tables using .load("warehouse/db/events"). Iceberg requires you to reference tables via catalogs using .table("catalog.db.table").

Best Practices

• Use Hidden Partitioning: Do not expose partition columns in your table schema manually; allow Iceberg to partition columns dynamically under the hood.

Interview Perspective

Interactive Challenges

Related Topics

• Delta Lake
• Apache Hudi
• Parquet