collect_list gathers the values of a column for each group into an array, keeping duplicates. collect_set does the same but keeps only distinct values.
Example
(orders.groupBy("customer_id")
.agg(F.collect_list("product").alias("all_products"),
F.collect_set("product").alias("distinct_products")))For customer 7 who bought pen, pen, book: all_products = [pen, pen, book], distinct_products = [pen, book].
The order trap
The order of elements is not guaranteed. Spark gathers rows from many partitions in whatever order they arrive, and the result can change between runs. If order matters, build it explicitly. Either sort the array afterwards with array_sort, or sort structs: collect struct(order_ts, product) and sort the array, which orders by the first field, then pull out the products. You can also use a window ordered by timestamp, with collect_list over it.
The memory trap
Each group becomes one array held in memory on a single task. If one customer has 50 million rows, that array is huge, and the task fails with an out-of-memory error. Before using these functions, estimate the biggest group. Often you do not need the list at all, and an aggregate such as count, max or approx_count_distinct does the job without gathering rows.
Working with the arrays
Higher-order functions run logic on array elements inside Spark's engine, so you do not need a Python UDF:
df.select(
F.transform("prices", lambda p: p * 1.18).alias("with_tax"),
F.filter("prices", lambda p: p > 100).alias("expensive"),
F.aggregate("prices", F.lit(0.0), lambda acc, p: acc + p).alias("total"),
)These live in pyspark.sql.functions from Spark 3.1 onwards. For older versions use the same functions through F.expr("transform(prices, p -> p * 1.18)").
In an interview
Say it in one line: collect functions are convenient but bring a whole group into one task, so use them only when groups are small, and never rely on their order.