Learn · design
Choosing an architecture under real constraints, and defending the choice.
How to design data systems at scale: batch vs streaming, 500M events/day case studies, cost analysis, and interview preparation. Diagrams and tradeoffs.
Core foundational modules are free. Advanced production modules need Pro.
Playable walkthroughs: watch the system move, predict the next step, stamp a memory seal, then practice. Completing a Trace counts toward readiness.
An interviewer says: 500 million events a day, dashboards no more than five minutes stale, and the finance team needs three years of history they can recompute. Every tool you know could be part of the answer. The job is not to name tools, it is to work out what the constraints force, what they leave open, what each option costs per month, and what breaks first when the volume triples. That reasoning is the difference between a mid-level and a senior data engineer.
This is the interview that decides your level and your offer. It is also the meeting where you either get the design you want or inherit someone else's. Both reward the same skill: making tradeoffs explicit instead of asserting preferences.
Working knowledge of the pieces
You need to know roughly what a warehouse, a queue, Spark, and a scheduler each do. This is the last track for a reason.
Recommended: batch and streaming basics
Most design questions are really a batch versus streaming tradeoff with cost attached.
5 stages, in the order they build on each other. Each stage lists the modules and lessons it covers, and what you should be able to do by the end of it.
Most people freeze because they start naming technologies. A method fixes that: clarify requirements, do the volume arithmetic, pick a shape, then justify each component against a constraint.
How to designFree
0/2
Read the brief before you draw boxes. Volume, SLA, late data, and consumers decide batch versus stream.
Module Checkpoint
2 conceptual questions to verify mastery
By the end of this stage
You have a repeatable order of attack for any design question, which mostly removes the panic.
Where people get stuck
Do the arithmetic out loud. 500 million events a day is about 5,800 per second, and that number decides most of the design. Candidates who skip it guess.
Work one large case properly, end to end, rather than skimming ten. The details are where the learning is.
500M events/day
0/3
Ingest, store, and fail at half a billion events a day. Fan-in, bronze prefixes, lakehouse formats, retries, DLQ, and SCD Type 2.
By the end of this stage
You can design ingestion, storage, and processing for a high volume event pipeline and say what each choice costs.
The last mile is usually ignored and usually where the money goes: how the data is served, how fresh it is, and what the monthly bill looks like.
Serving and cost
0/2
Who reads gold, how fresh it must be, and what the invoice looks like: warehouses, dbt marts, bytes scanned, retention, ephemeral Spark.
Module Checkpoint
1 conceptual questions to verify mastery
By the end of this stage
You can compare serving options on latency and cost, with numbers rather than adjectives.
Design interviews are a communication exercise as much as a technical one. Structure, checking assumptions, and stating tradeoffs are what get scored.
Interview
0/1
How to talk a design in 45 minutes: clarify, sketch, drill one path, name failure modes.
By the end of this stage
You can run a 45 minute design discussion with a clear structure and no dead air.
A full design you can present, defend, and revise when the interviewer changes a constraint.
Capstone
0/1
Walk 500M events/day end to end, naming the lesson from every other track where you practiced that hop.
By the end of this stage
You have one design you know so well that a follow-up question is welcome rather than threatening.
Finishing the lessons is not the goal. These are the things you should be able to do afterwards, and each one is worth checking honestly.
30 minutes a day
about 6 sessions
1 hour a day
about 3 sessions
4 hours a weekend day
about 1 session
Read less and talk more in this track. After each case, close the page and present the design out loud for ten minutes, ideally recorded. Design interviews are scored on how you explain, and reading silently trains none of that.
These counts cover reading and the built-in exercises only. Real practice on the drills and a capstone will add to it, and that time is where most of the learning happens.
| Common mistake | What to do instead |
|---|---|
| Naming a stack of tools as the answer. | Tools are the conclusion, not the argument. Start from requirements and volume, and let the components follow. |
| Designing for a scale nobody asked for. | Over-engineering is a real negative signal. Ask what the actual volume and freshness need is, and build for that with a note on how it scales. |
| Never mentioning cost. | Every senior design conversation involves money. Bringing it up unprompted is one of the cheapest ways to sound senior. |
| Practising by reading designs instead of presenting them. | The skill being tested is explanation under questions. Practise the thing being tested. |