Complex UDF with External Logic

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Data Quality. About 16 minutes. Part of the Pro drill bank.

Email addresses in user tables are often wrong. A quick data quality rule is to remove the ones that cannot be valid, for example because they have no @ sign. Some people write a Python UDF for this, but a built-in column check is much faster.

The table df has user_id and email. Two of the emails are clearly invalid: not-an-email and missing-at-sign. Neither has an @. Remove those rows and keep the others. Return user_id and email, sorted by user_id. The checker uses the two known bad values in this table, so you can filter by those 2 values. On a real cluster you would write a check like F.col("email").contains("@"), which works for any email, and not a Python UDF. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Remove the rows whose email is invalid (no @).
• Keep the rows with valid emails.
• Return user_id and email, sorted by user_id.
• Store the DataFrame in result.

Constraints

• Invalid emails in this table: not-an-email and missing-at-sign.
• Use a column expression, such as isin or contains, not a Python UDF.
• A real check would test for @, which handles values you have not seen yet.

Examples

Input: emails Output: user_id | email 1 | [email protected] 3 | [email protected] 5 | [email protected] Invalid fixtures are dropped.

Topics: lakebench, pyspark, email, filter.

More PySpark interview questions · All interview problems · Learn data engineering