UDF with Multiple Columns

PySpark data engineering interview problem. Difficulty: intermediate. Pattern: Filtering. About 16 minutes. Part of the Pro drill bank.

Build full_name with concat_ws instead of a multi-arg UDF. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

You often need to combine several columns into one, for example a first name and a last name into a full name. A beginner might write a UDF (a Python function) for this, but Spark has a built-in function that is faster: concat_ws, which joins columns with a separator you choose. Add a column full_name that joins first_name and last_name with a single space between them. Keep all 3 columns, first_name, last_name and full_name, and sort by first_name. The editor creates the table df and ends with result = df. Change the last line to build your answer.

Requirements

• Add full_name using F.concat_ws(" ", first_name, last_name).
• Keep first_name, last_name and full_name.
• Do not change the case of the letters.
• Sort by first_name and store the DataFrame in result.

Constraints

• The separator is one space.
• The names stay in lowercase, as in the input.
• Do not use a UDF. A built-in function does the job.

Examples

Input: names Output: first_name | last_name | full_name ada | lovelace | ada lovelace alan | turing | alan turing grace | hopper | grace hopper Native concat avoids UDF serialization.

Topics: lakebench, pyspark, concat_ws.

More PySpark interview questions · All interview problems · Learn data engineering