Day 38Phase 08 · Project #1 - E-Commerce Data Platform

Project #1, part 3: orchestrate, test, explain

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Assemble extraction, transformation, dbt modeling, and testing into an automated Airflow DAG.
  • Write a concise 10-minute project walkthrough explaining business problem, architecture, and trade-offs.
  • Defend failure recovery and idempotency mechanisms against common interviewer challenges.

The Master DAG: Orchestrating the Medallion Pipeline

In Parts 1 and 2, you built the individual extraction, curation, and modeling components. Today, we assemble them into an automated, production Airflow DAG:

Using Airflow Task Groups keeps the DAG legible and cleanly separated by tier:

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: End-to-end Airflow DAG tying extraction, Spark, and dbt together, Warehouse atomic MERGE / COPY ingestion gates, dbt data tests as blocking deployment gates, SLA monitoring & alert routing, Complete system architecture verbal defense, Cost & performance trade-off justification

Build: Platform Stage 3: End-to-end Airflow master DAG orchestrating extraction, Spark, warehouse COPY/MERGE, dbt tests, and alert notifications.

In an interview, walk through this platform using the 'Requirements → Bottlenecks → Trade-offs → Failure Modes' framework.

Already have a pass? Go to your account. Need a pass? See pricing.