DataPlayArena
SPARK Reference Guide
Revision Time: 9 mins

Read & Write APIs Reference

Comprehensive file reader & writer configurations for CSV, JSON, Parquet, ORC, Delta Lake, and JDBC databases.

spark.read.csv (Detailed Config)
Return: DataFrame

Configures and reads CSV datasets with explicit options for headers, schema inferencing, delimiters, and mode.

Used In: Raw File Ingestion, Log Parsing

Syntax signature:spark.read.format('csv')\ .option('header', 'true')\ .option('inferSchema', 'true')\ .option('delimiter', ',')\ .option('mode', 'PERMISSIVE')\ .option('nullValue', 'NA')\ .load(path)
Code snippet:
python
from pyspark.sql.types import StructType, StructField, StringType, IntegerType

custom_schema = StructType([
    StructField('id', IntegerType(), True),
    StructField('name', StringType(), True)
])

df = spark.read.format('csv')\
    .option('header', 'true')\
    .schema(custom_schema)\
    .load('data/*.csv')
Expected Output:Parsed DataFrame matching custom_schema
Common Mistakes:Using `inferSchema=True` on 500GB CSV files, causing 2-hour schema detection delays before job start.
Remember: CRITICAL PRODUCTION TIP: `inferSchema=True` scans the entire CSV dataset TWICE before returning. In production ETL, ALWAYS provide an explicit `schema(StructType(...))` for 10x faster reads.
spark.read.json
Return: DataFrame

Reads JSON documents (JSON Lines format where each line is a valid JSON object) or multiline JSON arrays.

Used In: API Response Ingestion, Event Log Processing

Syntax signature:spark.read.option('multiLine', 'true').json(path)
Code snippet:
python
df = spark.read\
    .option('multiLine', 'true')\
    .option('pruneUrl', 'true')\
    .json('records.json')
Expected Output:DataFrame with infered struct/array column types
Remember: Standard `spark.read.json()` expects 1 JSON object per line (JSONL). If reading a pretty-printed single file array `[{...}, {...}]`, set `.option('multiLine', 'true')`.
spark.read.parquet / ORC
Return: DataFrame

Reads columnar Parquet or ORC files with automatic metadata inspection, schema merging, and filter pushdown.

Used In: Data Lakes, Analytical Data Warehouses

Syntax signature:spark.read.option('mergeSchema', 'true').parquet('path/*.parquet')
Code snippet:
python
df = spark.read.parquet('s3a://my-bucket/events/year=2024/')
Expected Output:High-speed columnar DataFrame
Remember: Parquet stores column statistics (min/max values per row-group). Spark uses this metadata to SKIP entire row-groups during `filter()` scans (Data Skipping).
spark.read.jdbc (Relational DB)
Return: DataFrame

Connects to external SQL databases (PostgreSQL, MySQL, Oracle, SQL Server) over JDBC.

Used In: RDBMS Data Extraction, Data Warehouse Ingestion

Syntax signature:spark.read.format('jdbc')\ .option('url', 'jdbc:postgresql://host:5432/dbname')\ .option('dbtable', 'orders')\ .option('user', 'admin')\ .option('password', 'secret')\ .option('partitionColumn', 'order_id')\ .option('lowerBound', '1')\ .option('upperBound', '1000000')\ .option('numPartitions', '20')\ .load()
Code snippet:
python
jdbc_df = spark.read.format('jdbc')\
    .option('url', 'jdbc:mysql://localhost:3306/prod')\
    .option('dbtable', '(SELECT id, amount FROM orders WHERE status="PAID") as filtered_orders')\
    .option('user', 'root')\
    .option('password', 'pass')\
    .load()
Expected Output:DataFrame fetched from relational DB
Common Mistakes:Omitting `numPartitions` on JDBC reads, causing single-thread bottle-necks on 100M-row database tables.
Remember: CRITICAL PERFORMANCE TIP: Without `partitionColumn`, `lowerBound`, `upperBound`, and `numPartitions`, Spark reads the ENTIRE JDBC table through 1 single thread/partition! Always supply partition parameters for parallel JDBC reads.
df.write.mode & format
Return: None

Persists DataFrame contents to storage disk with targeted save modes and compression codecs.

Used In: ETL Output Persistence, Data Lake Storage

Syntax signature:df.write.mode('overwrite|append|ignore|errorifexists')\ .format('parquet')\ .option('compression', 'snappy|gzip|zstd')\ .save(path)
Code snippet:
python
df.write.mode('overwrite')\
    .format('parquet')\
    .option('compression', 'snappy')\
    .save('/mnt/output/clean_data')
Expected Output:Saved Parquet files in directory
Remember: Save modes: - `overwrite`: Replaces existing target directory. - `append`: Appends new files to target directory. - `ignore`: Silently skips write if directory exists. - `errorifexists`: Throws AnalysisException if directory exists.
df.write.partitionBy
Return: None

Partitions output files into sub-directories based on column values (`/year=2024/month=08/`).

Used In: Data Lake Organization, Partition Pruning

Syntax signature:df.write.partitionBy('year', 'month').parquet(path)
Code snippet:
python
df.write.mode('append')\
    .partitionBy('country', 'status')\
    .parquet('s3a://data-lake/orders/')
Expected Output:Folder structure: orders/country=US/status=PAID/part-000.parquet
Common Mistakes:Partitioning by `timestamp` column resulting in 1 million 2KB files.
Remember: WARNING: Do NOT partition by high-cardinality columns (e.g., timestamp, user_id, uuid). High cardinality creates 500,000+ tiny files (Small File Problem), degrading metastore and read speed. Aim for partition folder sizes between 100MB - 1GB.
df.write.format('delta')
Return: None

Writes DataFrame to Delta Lake storage format with ACID transaction guarantees and schema enforcement.

Used In: Lakehouse Architecture, Delta Tables, Medallion Architecture (Bronze/Silver/Gold)

Syntax signature:df.write.format('delta')\ .mode('append')\ .option('mergeSchema', 'true')\ .save(path)
Code snippet:
python
df.write.format('delta')\
    .mode('overwrite')\
    .option('overwriteSchema', 'true')\
    .save('/delta/events_table')
Expected Output:Delta Lake table with `_delta_log/` transaction log
Remember: Delta Lake provides ACID transactions, time-travel, upserts (MERGE INTO), and automatic small file compaction (`OPTIMIZE`).
Previous TopicSpark SQLNext TopicAggregations