Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

Write partitioned Parquet

Mediumparquetpartitionbyiolakehouse

Interview question

How do you write a partitioned Parquet dataset in PySpark?

How to answer

The short answer

Use partitionBy on the writer so Spark lays out Hive-style directories. Readers can then prune they do not need.

Python
(
    df.write
    .mode("overwrite")
    .partitionBy("event_date", "region")
    .option("compression", "snappy")
    .parquet("s3://lake/silver/events/")
)
# Creates:
# s3://lake/silver/events/event_date=2024-01-01/region=east/part-....parquet

Diagram

Output
DataFrame
   |
 write.partitionBy(event_date, region)
   |
 root/
   event_date=2024-01-01/region=east/...
   event_date=2024-01-01/region=west/...
   event_date=2024-01-02/region=east/...

Best practices

  • Choose low-to-medium partition columns (date is classic; user_id is usually too high).
  • Avoid thousands of tiny files per partition: control output partition count (repartition/coalesce, maxRecordsPerFile).
  • Prefer Parquet/ORC over CSV for analytical lakes.
  • For dynamic overwrite of only some partitions, configure partition overwrite mode when needed.
Python
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")

Read side

Python
spark.read.parquet("s3://lake/silver/events/").filter("event_date = '2024-01-01'")  # partition pruning

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill