Day 36Phase 08 · Project #1 - E-Commerce Data Platform

Project #1, part 1: ingest

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Extract incremental batches from PostgreSQL / REST APIs using watermark state tracking.
  • Enforce raw immutability by persisting unmutated payloads into partitioned bronze Parquet storage.
  • Validate payload schemas and quarantine malformed records into dead-letter directories.

Project #1 Architecture & Local Repository Structure

In Project #1, you build an end-to-end batch data platform for ShopKart e-commerce. You will run the complete pipeline on your laptop with zero cloud costs using Docker Compose and local storage, while mastering the exact architectural patterns used in high-scale cloud platforms:

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: Transactional database batch extraction, High-watermark state persistence, Partitioned Bronze raw lake staging, Schema validation & payload checksums, REST API event extraction, Idempotent extraction chunking

Build: Platform Stage 1: PostgreSQL batch extractor → checksum validator → partitioned Hive Parquet Bronze lake writer.

Never perform transformations during the raw extraction phase. Write Bronze data in its raw, unmodified form to enable historical replay.

Already have a pass? Go to your account. Need a pass? See pricing.