Mask PII and extract email domains

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Data Quality. About 16 minutes. Part of the Pro drill bank.

Extract the email domain, mask all but the last four phone digits and hash the email. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

customers has id, email and phone, both of which can be NULL. Return: id domain: the part of the email after @, in lowercase phone_masked: the phone number with every digit replaced by * except the last four digits; other characters (dashes) stay email_hash: the SHA-256 hash (hex string) of the lowercase email A NULL input gives a NULL output. Assign the DataFrame to result.

Requirements

• NULL stays NULL.
• Hash the lowercase email.

Constraints

• Phone numbers contain digits and possibly dashes.
• Hash is a 64 character hex string.

Examples

Input: customers id | email | phone 1 | Ann.Lee@Example.com | 9876543210 2 | bo@mail.co.uk | 555-123-4567 3 | NULL | 1234567890 4 | cy@Example.com | NULL Output: id | domain | phone_masked | email_hash 1 | example.com | ****3210 | b7e0d8372a47f54bbefeb251ab9ac1e9e6b2d1983263de10196101be3961ed23 2 | mail.co.uk | *-*-4567 | a8020a1ba920a63f7594994158b9003e0a99847bbdca6f78e129c455ea23447d 3 | NULL | ****7890 | NULL 4 | example.com | NULL | c42f5d0033a838d1fd7175a5c0a93acae479330b37bfd307e7fbe62ffae16029 Ann and Cy share the domain example.com (lowercased). The dashes in 555-123-4567 stay and only the last four digits remain visible. NULL email or phone gives NULL.

Topics: lakebench, pyspark, regexp_extract, regexp_replace, sha2, pii.

More PySpark interview questions · All interview problems · Learn data engineering