Read & Write APIs Reference

Ingest and persist structured files using built-in read and write configurations.

read.csv

Return: DataFrame

Loads a CSV file and parses columns.

Syntax signature:spark.read.csv(path, header=True, inferSchema=True)

Code snippet:

python

spark.read.csv('data.csv', header=True)

Remember: inferSchema triggers an extra job to scan files. For production, define schema explicitly.

write.parquet

Return: None

Saves DataFrame content as compressed Parquet files.

Syntax signature:df.write.mode('overwrite').parquet(path)

Code snippet:

python

df.write.mode('overwrite').parquet('output.parquet')

Remember: Parquet stores column blocks with metadata statistics, allowing projection and filter pushdowns.

write.partitionBy

Return: DataFrameWriter

Partitions the output layout by columns on storage disks.

Syntax signature:df.write.partitionBy(*cols)

Code snippet:

python

df.write.partitionBy('country').csv('dir')

Remember: Avoid partitioning by columns with high cardinality (e.g. timestamp/ID) to prevent creating millions of tiny files.