Phase 04 · Modern data platforms · python

Infrastructure & Engineering Practices

The engineering habits and tools that surround the data work.Git, Docker, Kubernetes, Terraform, and CI/CD explained from scratch with hands-on critique exercises. The tools run on your laptop; the concepts run here.
Start lesson 1The terminal in 15 minutes
lessons
13
modules
5
stages
5
of learning
2h 56m

Why this track exists

Your pipeline works on your laptop. It fails on the server because the Python version differs. Two people edit the same file and one person's work disappears. A change goes to production on Friday and nobody can say what changed or how to undo it. None of these are data problems, and all of them will happen to you. Git, containers, and CI exist because software written by more than one person needs them.

These tools are assumed rather than taught on the job. Nobody sits a new hire down and explains merge conflicts, they just expect the pull request to be clean. Being comfortable here is the difference between contributing in week one and being blocked by the toolchain.

What you need before starting

  • Some code you have written

    The concepts are much easier to grasp with a real script to version and containerise.

  • A terminal you are not afraid of

    The Linux module covers paths, pipes, and exit codes from scratch, so a little fear is fine.

The roadmap

5 stages, in the order they build on each other. Each stage lists the modules and lessons it covers, and what you should be able to do by the end of it.

01Git and the command line

Git confuses people because they learn commands without the model. Three places hold your work: the files you edit, the staging area, and the commit history. Once that is clear, every command makes sense.

Terminal, Git and Docker basicsFree

0/6

The terminal, working tree, index, commit, branches and pull requests, conflict resolution, PATH, pipes, exit codes, and running a first Docker container. No Git remote or Docker lives in this tab.

  1. The terminal in 15 minutes14m
  2. Git: working tree, index, commit12m
  3. Git with a team: branch, push, pull request14m
  4. Merge conflicts14m
  5. PATH, pipes, and exit codes12m
  6. Docker: run a container, then use Compose16m

By the end of this stage

You can branch, commit, and resolve a merge conflict as an ordered procedure rather than a panic, and you can read paths, pipes, and exit codes in a shell.

Where people get stuck

Exit codes matter more than they look. A pipeline step that fails but exits zero is a silent failure that a scheduler will happily call success.

02Containers

'It works on my machine' is a real, expensive problem. A container is the answer: ship the environment with the code.

Docker

0/2

Read a Dockerfile, spot secrets and a root USER. There is no Docker daemon in this tab. Build on your laptop.

  1. Read a Dockerfile14m
  2. Dockerfile bugs: USER root12m

By the end of this stage

You can read a Dockerfile, say what each layer does, and explain why image size and build order matter.

03Kubernetes and Terraform

Two tools people name-drop constantly. You mostly need to know what problem each solves and how to read what someone else wrote, not to administer a cluster.

Kubernetes and Terraform

0/2

CronJob versus Deployment, then a missing Terraform resource. No cluster and no terraform apply in this tab.

  1. Deployment vs CronJob14m
  2. Terraform: the missing resource14m

By the end of this stage

You can explain what a scheduler for containers is for, and what it means to describe infrastructure as code you can review and roll back.

04CI/CD

This is where quality gets enforced automatically: tests run, checks pass, and nobody deploys straight from a laptop.

CI/CD

0/2

Lint, test, build, deploy on main. GitHub Actions runs on GitHub, not in this tab.

  1. CI/CD stages12m
  2. CI order on main12m

By the end of this stage

You can describe a pipeline that tests and deploys a data job, and say what should block a merge.

05Capstone

Critique exercises on realistic setups, because reading someone else's infrastructure is the actual daily skill.

Capstone

0/1

Order the full ship: commit, image, CI, terraform plan, deploy. Plan stays in the list. Apply stays on your account.

  1. Capstone: ship checklist16m

By the end of this stage

You can review a Dockerfile or a CI config and point at the real problems.

Finish this track, get a certificate

Complete all 13 lessons and claim a certificate with a link you can add to LinkedIn or send to a recruiter.

How certificates work

Mistakes to avoid on this track

Common mistakes on this track and what to do instead
Common mistakeWhat to do instead
Learning Git as a list of commands to copy when things break.Learn the three areas and the history graph. Then the commands are obvious and recovery is calm.
Trying to learn Kubernetes deeply as a data engineer.Know what it does and how to read a manifest. Deep cluster administration is a different job.
Committing secrets or config to the repo.Config belongs in the environment. Once a secret is in history it stays there until someone rewrites it.

Where to practise this

Where to go after this

Infrastructure & Engineering Practices reviews & rating

Finished a few lessons? Tell us what worked and what was confusing. Reviews are read by hand and help decide what gets rewritten next.