Write partitioned Parquet

Use partitionBy on the writer so Spark lays out Hive-style directories. Readers can then prune partitions they do not need.

(
    df.write
    .mode("overwrite")
    .partitionBy("event_date", "region")
    .option("compression", "snappy")
    .parquet("s3://lake/silver/events/")
)
# Creates:
# s3://lake/silver/events/event_date=2024-01-01/region=east/part-....parquet

Diagram

DataFrame
   |
 write.partitionBy(event_date, region)
   |
 root/
   event_date=2024-01-01/region=east/...
   event_date=2024-01-01/region=west/...
   event_date=2024-01-02/region=east/...

Best practices

• Choose low-to-medium cardinality partition columns (date is classic; user_id is usually too high).
• Avoid thousands of tiny files per partition: control output partition count (repartition/coalesce, maxRecordsPerFile).
• Prefer Parquet/ORC over CSV for analytical lakes.
• For dynamic overwrite of only some partitions, configure partition overwrite mode when needed.

spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

Read side

spark.read.parquet("s3://lake/silver/events/").filter("event_date = '2024-01-01'")  # partition pruning