All roadmaps

Data engineer

Data engineer path

From SQL and Python to pipelines, Spark and the cloud, in the order a first data engineering job asks for them.

  • about 18 weeks at an hour a day
  • 5 stages, 37 steps
Start step 1See the roadmap

SQL you can rely on

about 4 weeks · 7 steps

After this stage: You can write joins, CTEs and window functions without looking them up.

  1. What is data engineering?LessonFreeWhat the job is before the tools.01
  2. Aggregations and groupingLessonLockedGROUP BY and HAVING.02
  3. Multi-table joinsLessonLockedWhat each join keeps.03
  4. Subqueries and CTEsLessonLockedBreak a hard query into steps.04
  5. Window functionsLessonLockedThe data engineering benchmark.05
  6. Deduplication and incremental upsertsLessonLockedLoad only what changed.06
  7. SQL 50 practice sheetPractice50 ordered problems.07

Python for pipelines

about 4 weeks · 9 steps

After this stage: You can write a script that reads, cleans and loads data.

  1. Why Python for data engineers?LessonFreeWhat Python is used for here.08
  2. Lists, dictionaries and setsLessonLockedThe containers behind every record.09
  3. File I/O and data formatsLessonLockedCSV, text and Parquet basics.10
  4. Working with JSONLessonLockedThe format of most APIs.11
  5. Working with REST APIsLessonLockedPull data from a service.12
  6. Retries, backoff and idempotencyLessonLockedMake a rerun safe.13
  7. Python 100 practice sheetPractice100 ordered problems.14
  8. Pandas: what is pandas?LessonFreeTables in Python.15
  9. Data cleaning and type castingLessonLockedFix the data before you load it.16

Pipelines and orchestration

about 3 weeks · 7 steps

After this stage: You can describe a pipeline, schedule it and handle a rerun.

  1. What is a data pipeline?LessonFreeSources, steps and destinations.17
  2. Scheduling and cronLessonFreeRun a job on a timetable.18
  3. Idempotent loadsLessonLockedRun twice, get the same result.19
  4. Backfills and catchupLessonLockedFill in the past safely.20
  5. CDC: inserts, updates, deletesLessonLockedCapture what changed.21
  6. Case study: load only what changedCase studyAn incremental load on real-looking data.22
  7. Case study: do the two systems agree?Case studyReconcile a source with its copy.23

Warehouse, Spark and quality

about 4 weeks · 8 steps

After this stage: You can model a star schema, use Spark and add checks.

  1. OLTP vs OLAP mental modelLessonLockedWhy warehouses are different.24
  2. Star schema: facts, dimensions, grainLessonLockedThe shape of analytics data.25
  3. PySpark architecture: the one explanationLessonFreeHow Spark runs your code.26
  4. DataFrame basicsLessonLockedRead, filter and select in Spark.27
  5. Joins and optimization strategiesLessonLockedJoins at scale.28
  6. What is data quality?LessonFreeWhy checks belong in the pipeline.29
  7. Case study: which events are broken?Case studyFind bad records before they spread.30
  8. PySpark 40 practice sheetPracticePractise the Spark moves.31

Cloud, projects and interviews

about 3 weeks · 6 steps

After this stage: You have projects to show and answers for the system design round.

  1. What is cloud computing?LessonFreeThe vocabulary you need.32
  2. Data lakes versus warehousesLessonFreeWhere each one fits.33
  3. ProjectsProjectBuild a pipeline end to end.34
  4. 45-day data engineer planToolA daily plan if you want a fixed schedule.35
  5. Data engineer roadmapToolThe full list of topics and what to learn when.36
  6. Mock interviewToolPractise out loud and get scored.37

After this roadmap