On this page
Big data roles often use Apache Spark, and PySpark is Spark with Python. These roles pay attention to how work is spread across machines. They can seem intimidating, but the entry-level questions mostly check the basics. If you know SQL and Python, you are already part of the way there.
Joins, GROUP BY and window functions.
DataFrames and cleaning.
Partitions, transformations and actions.
DataFrames, joins, aggregations.
Shuffles, skew and partition counts.
What Spark is for
When data is too big for one computer, you split it into parts and let many computers work on the parts at once. Spark is a tool that does that splitting and combining for you. Read what is Apache Spark for a plain explanation. You usually write DataFrame code that looks like SQL or pandas, and Spark decides how to run it.
What to learn, in order
- SQL and Python. Spark SQL is SQL, and PySpark is Python. Strong basics here make Spark much easier.
- DataFrames. Select, filter, join, group and aggregate with PySpark. See the PySpark and distributed track.
- Lazy work. Transformations plan the work, and actions run it. This idea appears in many interviews.
- Partitions and shuffles. See shuffle partitions explained.
- Skew. See data skew explained.
- File formats. Parquet vs CSV and why. See CSV vs JSON vs Parquet.
How to practise without a cluster
You can install PySpark on a laptop and run it in local mode with small files. The concepts are the same. Use a few thousand to a million rows of public data. Write the same task in SQL, pandas and PySpark and compare. Practise on the PySpark practice page, which runs in your browser.
What interviews ask
- What is the difference between a transformation and an action?
- What is a partition, and how does the number of partitions affect speed?
- What is a shuffle, and why is it costly?
- A job is slow because one key has most of the data. What do you do?
- When would you use Parquet instead of CSV?
- Write a PySpark query for the top 3 products per category.
See PySpark scenario-based interview questions for worked answers.
A project to show
Take a larger public dataset, such as taxi trips or web logs. Load it with PySpark, clean it, aggregate it, and write the result as Parquet partitioned by date. Compare the run time with a pandas version on the same data. Write down what you learned about partitions. A short README with a diagram of the flow is enough.
Be honest about experience
Do not claim to have run Spark at huge scale if you have not. It is fine to say you practised on a laptop with public data and that you understand how the ideas scale. Interviewers trust honest answers, and they ask follow-up questions on every claim.
Where to start today
Solve one PySpark practice problem. Then write the same query in SQL and compare the two.