Day 37Phase 08 · Project #1 - E-Commerce Data Platform

Project #1, part 2: transform

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Clean, cast, and deduplicate bronze data into silver tables using PySpark window functions.
  • Build gold star schema dimensions and aggregated daily sales facts with broadcast joins.
  • Run reconciliation audits verifying that gold GMV totals match bronze transactional sums.

The Medallion Progression: Bronze to Silver to Gold

In Part 1, you landed raw operational database extracts as immutable Parquet files in the Bronze layer. Now, we transform those raw payloads into clean, trusted data assets for business reporting:

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: Medallion curation: Bronze raw to Silver cleaned to Gold aggregates, Windowed deduplication by primary key and updated_at timestamp, Data contract casting & null substitution policies, Broadcast dimension enrichment, Gold daily revenue fact tables, Atomic S3 partition swaps

Build: Platform Stage 2: PySpark Silver deduplication & validation engine → Gold metric aggregations → broadcast join enricher → atomic lakehouse writer.

Always deduplicate Silver data using row_number() over (partition by id order by updated_at desc) before performing any join operations.

Already have a pass? Go to your account. Need a pass? See pricing.