intermediate

Hive

By DataPlayArena Technical Review BoardUpdated 2026-07-088 min readRuntime Verified

Overview

Connect Spark to Apache Hive Metastore to share table metadata, write catalog tables, and query partitioned Hive tables.

What You Will Learn

In this lesson, you will learn:
  • Hive Metastore: Reading shared table schemas across SQL platforms.
  • Enable Hive Support: Configuring SparkSession with Hive support.
  • Catalog tables: Interacting with managed vs external Hive tables.

Detailed Concept Explanation

By default, Spark uses an in-memory catalog to track temporary tables and views. To persist table schemas across different sessions and share them with other engines (like Presto or Impala), you can connect Spark to a shared Hive Metastore.


Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

# Initialize SparkSession with Hive Support enabled
spark = SparkSession.builder \
    .appName("HiveTest") \
    .enableHiveSupport() \
    .getOrCreate()

# Create a managed Hive table
spark.sql("CREATE TABLE IF NOT EXISTS hive_users (id INT, name STRING) USING hive")

# Insert data
spark.sql("INSERT INTO hive_users VALUES (1, 'Alice')")

# Query table
spark.sql("SELECT * FROM hive_users").show()

Expected Output

text
+---+-----+
| id| name|
+---+-----+
|  1|Alice|
+---+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
enableHiveSupport()
spark.sql(CREATE TABLE)
spark.sql(INSERT INTO)
spark.sql(SELECT)
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder()
  .appName("HiveScala")
  .enableHiveSupport()
  .getOrCreate()

spark.sql("CREATE TABLE IF NOT EXISTS hive_users (id INT, name STRING) USING hive")
spark.sql("INSERT INTO hive_users VALUES (1, 'Alice')")

spark.sql("SELECT * FROM hive_users").show()

Expected Output

text
+---+-----+
| id| name|
+---+-----+
|  1|Alice|
+---+-----+

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.