Day 36Phase 08 · Project #1 - E-Commerce Data Platform
Project #1, part 1: ingest
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Extract incremental batches from PostgreSQL / REST APIs using watermark state tracking.
- Enforce raw immutability by persisting unmutated payloads into partitioned bronze Parquet storage.
- Validate payload schemas and quarantine malformed records into dead-letter directories.
Project #1 Architecture & Local Repository Structure
In Project #1, you build an end-to-end batch data platform for ShopKart e-commerce. You will run the complete pipeline on your laptop with zero cloud costs using Docker Compose and local storage, while mastering the exact architectural patterns used in high-scale cloud platforms:
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Transactional database batch extraction, High-watermark state persistence, Partitioned Bronze raw lake staging, Schema validation & payload checksums, REST API event extraction, Idempotent extraction chunking
Build: Platform Stage 1: PostgreSQL batch extractor → checksum validator → partitioned Hive Parquet Bronze lake writer.
Never perform transformations during the raw extraction phase. Write Bronze data in its raw, unmodified form to enable historical replay.
Already have a pass? Go to your account. Need a pass? See pricing.