Solve “Broadcast Join”

Free. Runs in your browser, no install.

PySpark · DataFrame API & I/O

dropDuplicates vs distinct

Mediumdedupedistinctdropduplicates

Interview question

What is the difference between dropDuplicates and distinct in PySpark?

How to answer

The short answer

Both remove duplicate rows, but `dropDuplicates` (alias `drop_duplicates`) can target a subset of columns, while `distinct()` dedupes on all columns.

Python
# All columns
df.distinct()
df.dropDuplicates()          # same idea when no subset given

# Subset of columns (keep one row per key)
df.dropDuplicates(["user_id", "order_date"])

Behavior notes

Output
distinct()
  -> equivalent to SELECT DISTINCT * ; shuffles

dropDuplicates([cols])
  -> one row per unique combination of cols
  -> which row is kept among ties is not a stable "business rule"
     unless you define order (window + filter)

Deterministic dedupe pattern

Python
from pyspark.sql import functions as F
from pyspark.sql.window import Window

w = Window.partitionBy("user_id", "event_id").orderBy(F.col("ingest_ts").desc())
deduped = (
    df.withColumn("rn", F.row_number().over(w))
      .filter(F.col("rn") == 1)
      .drop("rn")
)

Interview tip

Say when you need "latest record wins" you do not rely on dropDuplicates alone; you use a window ordered by a timestamp.

Lock it in with a real drill

You remember an answer longer once you have used it. Try a hands-on problem on this concept.

Open related drill