intermediate

Google Cloud Storage

By DataPlayArena Technical Review BoardUpdated 2026-07-0810 min readRuntime Verified

Overview

Connect Spark to Google Cloud Storage (GCS) using the GCS connector to read and write files directly from storage buckets.

What You Will Learn

In this lesson, you will learn:
  • GCS Connector: Configuring the GCS filesystem connector.
  • Credentials: Configuring service account keys.
  • Bucket Paths: Reading and writing data using the gs:// URI scheme.

Detailed Concept Explanation

Spark connects to Google Cloud Storage buckets using the GCS connector.

You must specify the path using the gs:// URI scheme.


Code Examples

Python (PySpark) Implementation

python
from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("GCSTest") \
    .config("spark.jars.packages", "com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.5") \
    .getOrCreate()

# Configure service account credentials on the Hadoop configuration
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
hadoop_conf.set("fs.gs.auth.service.account.enable", "true")
hadoop_conf.set("fs.gs.auth.service.account.json.keyfile", "/path/to/keyfile.json")

# Read Parquet files from a GCS bucket
df = spark.read.parquet("gs://my-bucket/data/logs.parquet")
df.show()

Expected Output

text
+---+-----+
| id|level|
+---+-----+
|  1| INFO|
+---+-----+

Execution Plan Diagram (Python & Scala)

Execution Plan Diagram
SparkSession.builder
hadoopConfiguration().set(fs.gs)
read.parquet(gs://my-bucket)
show()

Scala Implementation

scala
import org.apache.spark.sql.SparkSession

val spark = SparkSession.builder().appName("GCSScala").getOrCreate()

val sc = spark.sparkContext
sc.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
sc.hadoopConfiguration.set("fs.gs.auth.service.account.enable", "true")
sc.hadoopConfiguration.set("fs.gs.auth.service.account.json.keyfile", "/path/to/keyfile.json")

val df = spark.read.parquet("gs://my-bucket/data/logs.parquet")
df.show()

Expected Output

text
+---+-----+
| id|level|
+---+-----+
|  1| INFO|
+---+-----+

Related Topics

DPA

DataPlayArena Technical Editorial Board

Curated & maintained by enterprise data platform engineers with 10+ years in distributed systems.

Runtime Verification Standards: Every code example, query, and architecture diagram in this tutorial is compiled and verified against real runtimes (Apache Spark 3.5, Python 3.11, PostgreSQL, and DuckDB). If you spot any errata or want to suggest improvements, please submit feedback.