Distinct Count

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Aggregation. About 12 minutes. Free to practice.

Count distinct user_id values into a 1-row frame. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

You want to know how many different users appear in a table of events. A user can have many rows, but should be counted once. From the table df, count the number of distinct user_id values. Return a DataFrame with 1 row and 1 column called distinct_users. Save your final table in result. Let Spark do the counting, so do not collect the rows into Python.

Requirements

  • Count each user_id once, no matter how many rows it has.
  • Return a DataFrame with 1 row.
  • Name the column distinct_users.

Constraints

  • The input table is df.
  • The output must be a DataFrame, not a plain Python number.
  • Prefer distinct().count() or F.countDistinct, rather than collecting rows.

Examples

Input: events fixture Output: distinct_users 3 Users 1, 2, and 3 appear in the fixture.

Topics: lakebench, pyspark, distinct, count.

More PySpark interview questions · All interview problems · Learn data engineering

beginner

Distinct Count

Interview-style drill: Count distinct user_id values into a 1-row frame.

From `df`, compute the number of distinct `user_id` values. Return a 1-row DataFrame with column `distinct_users`. Assign `result`. Prefer distinct().count() over collecting rows.