Add Derived Column

PySpark data engineering interview problem. Difficulty: beginner. Pattern: Filtering. About 12 minutes. Free to practice.

Add full_name by joining first_name and last_name with a space. Treat this as a production helper: match the contracted return shape, including empty and duplicate inputs.

You have a table of people with a first_name and a last_name. A report needs 1 column with the full name, with a space between the first and last name. Add a new column called full_name that joins first_name and last_name with a single space, using the Spark function concat_ws. Keep all 3 columns (first_name, last_name, full_name). Sort by first_name. Save your final table in result.

Requirements

• Add a column named full_name.
• full_name is the first name, then one space, then the last name.
• Use F.concat_ws, not a Python loop.
• Keep first_name and last_name in the output.
• Sort by first_name.

Constraints

• The input table is df with the columns first_name and last_name.
• concat_ws(sep, col1, col2) joins columns using sep between them. Here the separator is a single space ' '.
• Use withColumn('full_name', ...) to add the column.

Examples

Input: first_name | last_name Output: first_name | last_name | full_name ada | lovelace | ada lovelace alan | turing | alan turing dennis | ritchie | dennis ritchie grace | hopper | grace hopper concat_ws inserts a single space between non-null parts.

Topics: lakebench, pyspark, withColumn, concat_ws.

More PySpark interview questions · All interview problems · Learn data engineering