intermediate

BigQuery

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Connect Spark to Google Cloud BigQuery using the BigQuery connector to read and write large-scale data warehouse tables.

What You Will Learn

In this lesson, you will learn:
  • BigQuery Connector: Loading the BigQuery connector dependency.
  • Authentication: Configuring GCP credentials.
  • DataFrame Operations: Reading and writing BigQuery datasets.

Detailed Concept Explanation

To connect Spark to BigQuery, you use Google Cloud's BigQuery connector for Spark. The connector writes intermediate data to a Google Cloud Storage (GCS) temporary bucket before loading it into BigQuery.


Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("BigQueryTest") \
    .config("spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.24.2") \
    .getOrCreate()

# Read from a BigQuery table
bq_df = spark.read.format("bigquery") \
    .option("table", "my-project.my_dataset.my_table") \
    .load()
bq_df.show()

Expected Output

text
+---+-----+
| id| name|
+---+-----+
|  1|Alice|
+---+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
read.format(bigquery)
option(table)
load()
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("BigQueryScala").getOrCreate()

val bqDF = spark.read.format("bigquery")
  .option("table", "my-project.my_dataset.my_table")
  .load()
bqDF.show()

Expected Output

text
+---+-----+
| id| name|
+---+-----+
|  1|Alice|
+---+-----+

Common Mistakes

  • Forgetting Temporary GCS Bucket: Forgetting to configure a temporary GCS bucket when writing data to BigQuery. The connector requires a staging bucket to store intermediate files. Set the option .option("temporaryGcsBucket", "my-temp-bucket").

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.