Handle NULL Values

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Missing Data. About 12 minutes. Free to practice.

Fill null emails, then drop rows with null phone. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Real data has missing values. In this table of users, some have no email and some have no phone number. Both are shown as NULL (None in Python). Clean the table in two steps. First, replace every missing email with the text [email protected]. Then remove the users who have no phone number. Return user_id, email and phone, sorted by user_id. Do the email step first, then remove the rows. Do not make up phone numbers. Save the final table in result.

Requirements

• Fill NULL values in email with [email protected].
• Remove every row where phone is NULL.
• Do not invent or fill phone numbers.
• Return only user_id, email and phone, sorted by user_id.

Constraints

• The input table is df with the columns user_id, email and phone.
• Use fillna({'email': ...}) for the email and dropna(subset=['phone']) for the phone.
• Fill the email first, then drop the rows.

Examples

Input: user_id | email | phone Output: user_id | email | phone 1 | [email protected] | 111 2 | [email protected] | 222 5 | [email protected] | 555 Rows 3 and 4 lose phone and are dropped; row 2 keeps the filled email.

Topics: lakebench, pyspark, fillna, dropna.

More PySpark interview questions · All interview problems · Learn data engineering