PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Data Quality. About 16 minutes. Part of the Pro drill bank.
Keep emails that look valid without a Python UDF. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.
Filter out known-invalid emails lacking @ (the fixture marks them as not-an-email and missing-at-sign). Return user_id, email ordered by user_id. Assign result. On a real cluster you would use a Column expression like email.contains('@') instead of a Python UDF.
Input: emails Output: user_id | email 1 | ada@x.com 3 | alan@lab.org 5 | ok@ok.io Invalid fixtures are dropped.
Topics: lakebench, pyspark, email, filter.
More PySpark interview questions · All interview problems · Learn data engineering
Interview-style drill: Keep emails that look valid without a Python UDF.
Filter out known-invalid emails lacking `@` (the fixture marks them as `not-an-email` and `missing-at-sign`). Return `user_id`, `email` ordered by `user_id`. Assign `result`. On a real cluster you would use a Column expression like `email.contains('@')` instead of a Python UDF.