Handle Missing Values

Pandas data engineering interview problem. Difficulty: beginner. Pattern: Missing Data. About 12 minutes. Free to practice.

Show dropna, mean fill, and ffill strategies labeled by method. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

Missing values (NaN) are normal in real data, and there are several ways to handle them. Each way gives a slightly different result, so it is good to compare them. From the DataFrame df (with user_id and score, where some scores are missing), create 3 versions of the table. In the first, drop every row that has a missing score. In the second, fill the missing scores with the mean (average) of the scores that exist. In the third, forward-fill, which means each missing score is replaced by the last known score above it. Add a column called method with the values drop, mean and ffill to say which version each row belongs to. Stack the 3 versions into one table and sort by method and then user_id. Save the final table in result.

Requirements

• Build 3 versions of df: dropped, mean-filled and forward-filled.
• Add a method column with the values drop, mean and ffill.
• Stack the 3 versions with pd.concat.
• Sort by method and then by user_id.

Constraints

• df has user_id and score. Users 2 and 4 have a missing score.
• The mean is computed from the scores that exist: (10 + 30 + 50 + 60) / 4 = 37.5.
• Forward-fill copies the previous known value down into the gap.

Examples

Input: scores with nulls Output: user_id | score | method 1 | 10.0 | drop 3 | 30.0 | drop 5 | 50.0 | drop 6 | 60.0 | drop 1 | 10.0 | ffill 2 | 10.0 | ffill 3 | 30.0 | ffill 4 | 30.0 | ffill 5 | 50.0 | ffill 6 | 60.0 | ffill 1 | 10.0 | mean 2 | 37.5 | mean 3 | 30.0 | mean 4 | 37.5 | mean 5 | 50.0 | mean 6 | 60.0 | mean Mean of non-null scores is 37.5; ffill carries prior values.

Topics: lakebench, pandas, fillna, dropna, ffill.

More interview problems · All interview problems · Learn data engineering