Phase 04 · Modern data platforms · python
Infrastructure & Engineering Practices
- lessons
- 13
- modules
- 5
- stages
- 5
- of learning
- 2h 56m
Why this track exists
Your pipeline works on your laptop. It fails on the server because the Python version differs. Two people edit the same file and one person's work disappears. A change goes to production on Friday and nobody can say what changed or how to undo it. None of these are data problems, and all of them will happen to you. Git, containers, and CI exist because software written by more than one person needs them.
These tools are assumed rather than taught on the job. Nobody sits a new hire down and explains merge conflicts, they just expect the pull request to be clean. Being comfortable here is the difference between contributing in week one and being blocked by the toolchain.
What you need before starting
Some code you have written
The concepts are much easier to grasp with a real script to version and containerise.
A terminal you are not afraid of
The Linux module covers paths, pipes, and exit codes from scratch, so a little fear is fine.
The roadmap
5 stages, in the order they build on each other. Each stage lists the modules and lessons it covers, and what you should be able to do by the end of it.
01Git and the command line
Git confuses people because they learn commands without the model. Three places hold your work: the files you edit, the staging area, and the commit history. Once that is clear, every command makes sense.
Terminal, Git and Docker basicsFree
0/6
The terminal, working tree, index, commit, branches and pull requests, conflict resolution, PATH, pipes, exit codes, and running a first Docker container. No Git remote or Docker lives in this tab.
By the end of this stage
You can branch, commit, and resolve a merge conflict as an ordered procedure rather than a panic, and you can read paths, pipes, and exit codes in a shell.
Where people get stuck
Exit codes matter more than they look. A pipeline step that fails but exits zero is a silent failure that a scheduler will happily call success.
02Containers
'It works on my machine' is a real, expensive problem. A container is the answer: ship the environment with the code.
Docker
0/2
Read a Dockerfile, spot secrets and a root USER. There is no Docker daemon in this tab. Build on your laptop.
By the end of this stage
You can read a Dockerfile, say what each layer does, and explain why image size and build order matter.
03Kubernetes and Terraform
Two tools people name-drop constantly. You mostly need to know what problem each solves and how to read what someone else wrote, not to administer a cluster.
Kubernetes and Terraform
0/2
CronJob versus Deployment, then a missing Terraform resource. No cluster and no terraform apply in this tab.
By the end of this stage
You can explain what a scheduler for containers is for, and what it means to describe infrastructure as code you can review and roll back.
04CI/CD
This is where quality gets enforced automatically: tests run, checks pass, and nobody deploys straight from a laptop.
CI/CD
0/2
Lint, test, build, deploy on main. GitHub Actions runs on GitHub, not in this tab.
By the end of this stage
You can describe a pipeline that tests and deploys a data job, and say what should block a merge.
05Capstone
Critique exercises on realistic setups, because reading someone else's infrastructure is the actual daily skill.
Capstone
0/1
Order the full ship: commit, image, CI, terraform plan, deploy. Plan stays in the list. Apply stays on your account.
By the end of this stage
You can review a Dockerfile or a CI config and point at the real problems.
Finish this track, get a certificate
Complete all 13 lessons and claim a certificate with a link you can add to LinkedIn or send to a recruiter.
Mistakes to avoid on this track
| Common mistake | What to do instead |
|---|---|
| Learning Git as a list of commands to copy when things break. | Learn the three areas and the history graph. Then the commands are obvious and recovery is calm. |
| Trying to learn Kubernetes deeply as a data engineer. | Know what it does and how to read a manifest. Deep cluster administration is a different job. |
| Committing secrets or config to the repo. | Config belongs in the environment. Once a secret is in history it stays there until someone rewrites it. |
Where to practise this
Where to go after this
Infrastructure & Engineering Practices reviews & rating
Finished a few lessons? Tell us what worked and what was confusing. Reviews are read by hand and help decide what gets rewritten next.