Zoya Joshi

@zoyajoshi

Free member

Data engineer at a a SaaS analytics team. Mostly Iceberg and streaming pipelines.

Ahmedabad, Gujarat0–2 years1 questions1 answers

Icebergdbt

Streak: 0d
XP: 255
Lessons done: 1
Studio passes: 2

First LessonFirst Ticket

Share on X Share on LinkedIn

Questions

votes

answers

views

Window function over an unbounded partition, performance cliff at scale

A running total using Window.partitionBy("customer_id").orderBy("date") worked fine in dev on 1M rows, then fell over on 500M rows in prod with massive shuffle spill. Anything to do besides get a bigger cluster?

PySparkpysparkwindow-functions