Word count

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Aggregation. About 12 minutes. Free to practice.

Word count is the first example in almost every big data course. It shows the idea that Spark is built on: split the work into small pieces (words), then count them together.

The table lines has 1 column, line, with raw text. Some lines are empty, and some have capital letters and punctuation such as commas and exclamation marks. Count how many times each word appears. Change the text to lowercase, remove the punctuation, split each line into words on whitespace, and ignore empty pieces. Return word and count, with the most frequent word first. Words with the same count are sorted alphabetically. The editor creates the table lines and ends with result = lines. Change the last line to build your answer.

Requirements

• Make the text lowercase and remove everything that is not a letter, digit or space.
• Split each line on whitespace and put each word on its own row (explode).
• Ignore empty words.
• Count each word, and sort by count descending, then word.
• Return the columns word and count in result.

Constraints

• Words are compared in lowercase, so Spark, SPARK and spark are the same word.
• Punctuation is removed before splitting, so data, becomes data.
• An empty line gives no words.

Examples

Input: lines line Spark makes big data simple Big data, big wins! SPARK spark Output: word | count big | 3 spark | 3 data | 2 makes | 1 simple | 1 wins | 1 spark appears 3 times once case is ignored, big 3 times (with the punctuation removed), data twice, and the empty line adds nothing.

Topics: lakebench, pyspark, word count, split, groupby.

More PySpark interview questions · All interview problems · Learn data engineering