intermediate
Google Cloud Storage
10 min readLast updated: 2026-07-08
Overview
Connect Spark to Google Cloud Storage (GCS) using the GCS connector to read and write files directly from storage buckets.
What You Will Learn
In this lesson, you will learn:
- GCS Connector: Configuring the GCS filesystem connector.
- Credentials: Configuring service account keys.
- Bucket Paths: Reading and writing data using the
gs://URI scheme.
Detailed Concept Explanation
Spark connects to Google Cloud Storage buckets using the GCS connector.
You must specify the path using the gs:// URI scheme.
Code Examples
Python (PySpark) Implementation
python
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("GCSTest") \
.config("spark.jars.packages", "com.google.cloud.bigdataoss:gcs-connector:hadoop3-2.2.5") \
.getOrCreate()
# Configure service account credentials on the Hadoop configuration
hadoop_conf = spark._jsc.hadoopConfiguration()
hadoop_conf.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
hadoop_conf.set("fs.gs.auth.service.account.enable", "true")
hadoop_conf.set("fs.gs.auth.service.account.json.keyfile", "/path/to/keyfile.json")
# Read Parquet files from a GCS bucket
df = spark.read.parquet("gs://my-bucket/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+
Execution Plan Diagram (Python & Scala)
Execution Plan Diagram
SparkSession.builder
hadoopConfiguration().set(fs.gs)
read.parquet(gs://my-bucket)
show()
Scala Implementation
scala
import org.apache.spark.sql.SparkSession
val spark = SparkSession.builder().appName("GCSScala").getOrCreate()
val sc = spark.sparkContext
sc.hadoopConfiguration.set("fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")
sc.hadoopConfiguration.set("fs.gs.auth.service.account.enable", "true")
sc.hadoopConfiguration.set("fs.gs.auth.service.account.json.keyfile", "/path/to/keyfile.json")
val df = spark.read.parquet("gs://my-bucket/data/logs.parquet")
df.show()
Expected Output
text
+---+-----+
| id|level|
+---+-----+
| 1| INFO|
+---+-----+