Day 38Phase 08 · Project #1 - E-Commerce Data Platform
Project #1, part 3: orchestrate, test, explain
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Assemble extraction, transformation, dbt modeling, and testing into an automated Airflow DAG.
- Write a concise 10-minute project walkthrough explaining business problem, architecture, and trade-offs.
- Defend failure recovery and idempotency mechanisms against common interviewer challenges.
The Master DAG: Orchestrating the Medallion Pipeline
In Parts 1 and 2, you built the individual extraction, curation, and modeling components. Today, we assemble them into an automated, production Airflow DAG:
Using Airflow Task Groups keeps the DAG legible and cleanly separated by tier:
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: End-to-end Airflow DAG tying extraction, Spark, and dbt together, Warehouse atomic MERGE / COPY ingestion gates, dbt data tests as blocking deployment gates, SLA monitoring & alert routing, Complete system architecture verbal defense, Cost & performance trade-off justification
Build: Platform Stage 3: End-to-end Airflow master DAG orchestrating extraction, Spark, warehouse COPY/MERGE, dbt tests, and alert notifications.
In an interview, walk through this platform using the 'Requirements → Bottlenecks → Trade-offs → Failure Modes' framework.
Already have a pass? Go to your account. Need a pass? See pricing.