Broken gold: salted country rollup dropped buckets

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Skew. About 25 minutes. Part of the Pro drill bank.

Most of the clickstream traffic comes from two countries, US and IN. In Spark, all rows of one key go to one task, so these two countries made a few tasks very slow. This is called data skew. An engineer on call added a salt (a random-looking number added to the key to spread the rows), but then filtered to salt == 0, so only about one eighth of the rows were counted.

The gold job should count events per country. To spread the busy countries over more tasks, the job adds a column salt with a value from 0 to 7. The shipped version then keeps only the rows where salt == 0 before counting, so every country count is about 8 times too small. Fix the job. Do not drop any salt bucket. Either finish the 2-step salted count (first count by country and salt, then add the partial counts by country), or use a plain groupBy("country") that gives the same result. Return country and events, ordered by country. Store it in result and do not use .collect() or .toPandas().

Requirements

  • Count every event. No salt bucket may be thrown away.
  • If you use a salt, count by country and salt first, then sum the partial counts by country.
  • Return country and events, ordered by country.
  • Store the DataFrame in result. Do not use .collect() or .toPandas().

Constraints

  • The input is the table ecommerce_events, read with spark.table.
  • The salt is F.abs(F.hash("event_id")) % 8, which gives a number from 0 to 7.
  • The final counts must equal the plain count of events per country.

Examples

Input: country | salt | rows_in_bucket US | 0 | 40 US | 1 | 38 US | 2 | 41 IN | 0 | 35 IN | 1 | 33 Output: -- buggy filter salt == 0 keeps only: country | events IN | 35 US | 40 -- correct combine (all salts): country | events IN | 68 US | 119 Salting spreads a hot key across partitions. Filtering to one salt throws the other buckets away. Phase 2 must sum every salt partial back to the business key.

Topics: pyspark, skew, salt, debugging.

More PySpark interview questions · All interview problems · Learn data engineering

intermediate

Broken gold: salted country rollup dropped buckets

Production ticket: A skew 'fix' salted the clickstream then kept only salt == 0, under-counting every country.

US and IN traffic dominate `ecommerce_events`. An on-call salted the frame to spread the hot keys, then accidentally **filtered to salt == 0** before aggregating. Gold country counts are roughly 1/8 of bronze. Fix the PySpark job: - count events per `country` as `events` - order by country - assign `result` - **no** `.collect()` / `.toPandas()` - do not drop salt buckets - either finish a two-phase salt (partial by country+salt, then sum by country) or use a plain `groupBy("country")` that matches the same grain The incident log has the Spark stage timeline and the dbt volume check.