Day 16 of 45 · Pro lesson

Data Lakes

The curriculum for this day stays visible. The lesson, code, and workspace unlock with Pro.

Teaches: lake vs warehouse vs lakehouse, Medallion architecture, Parquet vs CSV, Hive partitioning, partition pruning, small files compaction

Build: Medallion Data Lake engine: raw CSV → bronze standardization → silver partitioned Parquet → gold daily sales aggregate.

•Complete 15 drills on Medallion contracts, column projection pushdown, Hive paths, and file compaction.
•Run Parquet vs CSV benchmarks, partition pruning simulations, and small file diagnostic modules.
•Simulate 5 failure modes: unpartitioned data swamp, high-cardinality partition explosion, and bill spikes.
•Defend 10 senior interview questions and complete the storage blank-page review.

Never destroy raw data. Store columnar Parquet with coarse temporal partitions so engines skip 90%+ of files.

This section walks through the idea with a short example, then the trade-offs you should mention in an interview.

In practice you start from the raw rows, apply the transform step by step, and check the shape of the result before you move on.

A common mistake is to jump straight to the final query without naming the grain or the join keys that keep the result correct.

Once the core path works, you harden it for nulls, duplicates, and late data so the pipeline stays reliable under load.

The Pro write-up covers the full explanation, worked examples, and the code you can run in the studio.

# Locked example
result = transform(frame)
print(result.head())

Already on Pro? Go to your account. Need a pass? See pricing.