All roadmaps

PySpark developer

PySpark developer path

Use the SQL and Python you know on data too big for one machine.

  • about 15 weeks at an hour a day
  • 5 stages, 38 steps
Start step 1See the roadmap

Know the basics first

about 3 weeks · 9 steps

After this stage: You are comfortable with SQL queries and Python functions.

  1. SQL aggregations and groupingLessonLockedSpark SQL uses the same ideas.01
  2. SQL multi-table joinsLessonLockedThe same joins, bigger data.02
  3. Functions, modules and importsLessonLockedPython you will use daily.03
  4. What is pandas?LessonFreeA single-machine twin of a Spark DataFrame.04
  5. Subqueries and CTEsLessonLockedSpark SQL reads the same.05
  6. Window functionsLessonLockedThe same ranking ideas, on bigger data.06
  7. Lists, dictionaries and setsLessonLockedPython containers around every job.07
  8. Reading and writing filesLessonLockedFormats you will meet in Spark.08
  9. Working with JSONLessonLockedNested data starts here.09

Spark DataFrames

about 3 weeks · 10 steps

After this stage: You can read, transform and write data in PySpark.

  1. PySpark architecture: the one explanationLessonFreeHow Spark runs your code.10
  2. When do you need Spark?LessonFreeKnow when not to use it.11
  3. Reading and writing dataLessonFreeFormats and options.12
  4. SQL inside SparkLessonFreeRun SQL on a DataFrame.13
  5. DataFrame basicsLessonLockedSelect, filter and withColumn.14
  6. Aggregations and groupingsLessonLockedgroupBy and agg.15
  7. PySpark 40 practice sheetPracticePractise the moves.16
  8. What is a cluster?LessonFreeDriver, executors and tasks.17
  9. PySpark architecture and mental modelLessonLockedLazy evaluation and the plan.18
  10. Column operations and built-in functionsLessonLockedUse built-ins before UDFs.19

Joins, windows and nested data

about 3 weeks · 6 steps

After this stage: You can handle the tasks interviews and real jobs repeat.

  1. PySpark window functionsLessonLockedRank and compare rows.20
  2. Joins and optimization strategiesLessonLockedBroadcast and shuffle joins.21
  3. Handling complex and nested dataLessonLockedArrays, structs and JSON.22
  4. Partitioning, repartition and coalesceLessonLockedControl how data is split.23
  5. Medallion pipeline projectLessonLockedBronze, silver and gold in one build.24
  6. Capstone part 2: incremental MERGELessonLockedUpdate a table without rewriting it.25

Fast and production ready

about 3 weeks · 7 steps

After this stage: You can find a slow stage and fix it.

  1. Caching and persistenceLessonLockedWhen to keep data in memory.26
  2. Data skew detection and saltingLessonLockedWhy one task is slower than the rest.27
  3. Reading a Catalyst physical planLessonLockedSee what Spark decided.28
  4. Delta Lake: MERGE, time travel, ACIDLessonLockedReliable tables on a data lake.29
  5. UDFs in depth, and why to avoid themLessonLockedThe slow path and the fast one.30
  6. PySpark projectsProjectBroadcast, skew salting and Delta Lake merges.31
  7. Mock interviewInterviewPractise explaining a Spark design.32

Streaming and production

about 3 weeks · 6 steps

After this stage: You can run Spark as a reliable scheduled job and explain the design.

  1. Structured Streaming and watermarksLessonLockedProcess events as they arrive.33
  2. Unit testing data transformsLessonLockedTest the logic without a cluster.34
  3. Data lakes versus warehousesLessonFreeWhere Spark output lands.35
  4. Scheduling and cronLessonFreeRun a job on time.36
  5. Idempotent loadsLessonLockedRerun a job safely.37
  6. What is data quality?LessonFreeCheck the output before you publish it.38

After this roadmap