Quality gate: telemetry rows missing a reading

Pandas data engineering interview problem. Difficulty: beginner. Pattern: Missing Data. About 12 minutes. Free to practice.

A factory sends sensor readings into a table called df_telemetry (480 rows). A data quality check named 'not null on reading' has failed. A check like this only tells you 'yes, some values are missing'. Finance does not want a yes or no. They want to see the actual rows, so they can chase the devices that are broken.

From the DataFrame df_telemetry, find the rows where reading is missing (NULL or NaN). Return only these 4 columns, in this order: reading_id, device_id, plant and metric. Leave out reading, event_time and ingest_time. Sort by reading_id. Save your final table in result.

Requirements

  • Keep only the rows where reading is missing.
  • Return only reading_id, device_id, plant and metric, in that order.
  • Do not return the reading, event_time or ingest_time columns.
  • Sort by reading_id.

Constraints

  • The input is df_telemetry with 480 rows.
  • A real part of the table has missing readings. This is not an empty or made-up case.
  • Test for missing values with isna(). Do not compare with == None or == np.nan, because those never match.

Examples

Input: reading_id | device_id | plant | metric | reading R00008 | DEV-008 | chennai-east | temp_c | (null) R00009 | DEV-009 | pune-west | vibration_mm | 0.412 Output: reading_id | device_id | plant | metric R00008 | DEV-008 | chennai-east | temp_c R00008 has no reading and is kept; R00009 has a real value and is dropped.

Topics: data-quality, not-null, pandas.

More interview problems · All interview problems · Learn data engineering

beginner

Quality gate: telemetry rows missing a reading

Production ticket: expect_column_not_null on telemetry.reading - return the actual violating rows, not just a pass/fail.

A not-null gate on `df_telemetry["reading"]` is failing. Finance wants the actual rows, not just the boolean. Return the rows of `df_telemetry` where `reading` is null. Columns: `reading_id`, `device_id`, `plant`, `metric` (drop `reading`, `event_time`, `ingest_time`). Sort by `reading_id`. Assign `result`.