Drop Columns

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Filtering. About 16 minutes. Part of the Pro drill bank.

Wide tables can have dozens of columns, and most of them are not needed for a given report. Keeping only the columns you need makes the job faster and also prevents private data (like a phone number) from leaking into places where it should not be.

The wide table df has 6 columns: user_id, email, phone, c4, c5 and c6. Keep only user_id, email and phone, and drop the others. Sort by user_id. The editor creates the table df and ends with result = df. Change the last line to build your answer. You can use select with the names you want, or drop with the names you do not want.

Requirements

• Keep exactly user_id, email and phone.
• Remove c4, c5 and c6.
• Sort by user_id.
• Store the DataFrame in result.

Constraints

• The result has 3 columns and 3 rows.
• Prefer select with explicit names, because a new column in the source cannot sneak into the result.
• The values of the kept columns are not changed.

Examples

Input: wide users Output: user_id | email | phone 1 | [email protected] | 111 2 | [email protected] | 222 3 | [email protected] | 333 Extra columns removed.

Topics: lakebench, pyspark, select.

More PySpark interview questions · All interview problems · Learn data engineering