Day 37 of 45 · Pro lesson

Project #1: PySpark Silver & Gold Curations

The curriculum for this day stays visible. The lesson, code, and workspace unlock with Pro.

Teaches: Medallion curation: Bronze raw to Silver cleaned to Gold aggregates, Windowed deduplication by primary key and updated_at timestamp, Data contract casting & null substitution policies, Broadcast dimension enrichment, Gold daily revenue & customer lifetime value fact tables, Atomic S3 partition swaps

Build: Platform Stage 2: PySpark Silver deduplication & validation engine → Gold metric aggregations → broadcast join enricher → atomic lakehouse writer.

•Complete 15 drills on Silver cleaning, window deduplication, broadcast join thresholds, and Gold rollup metrics.
•Run 6 Python modules verifying Silver contract cleanliness, Gold revenue reconciliations, and skew handling.
•Simulate 5 production failures: dimension duplicate row explosion, skewed seller bottleneck, float precision rounding drift, and partition write crash.
•Defend 10 senior Spark curation questions and complete the blank-page exam.

Always deduplicate Silver data using row_number() over (partition by id order by updated_at desc) before performing any join operations.

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

Already on Pro? Go to your account. Need a pass? See pricing.