Projects
Start with a beginner project and work up. Each one runs in your browser, with checks that tell you when it works.
These lessons are not a second catalog or a video course, and this tab cannot publish a repo for you. Passing every LakeBench lesson means you can write the transforms, the DAG habits, the tests, and the design argument. It does not publish a repo. The Cloud capstone is a numbered GCP free-tier checklist this tab cannot execute. Do those steps outside, then copy the solutions you wrote, add a README that states grain, schedule, and failure modes, and push that.
Ingest a CSV-shaped API feed, model a star from orders, and clean messy bronze into silver.
PySpark & Distributed Compute
PYSPARKTokenize distributed server log files, strip punctuation, filter stopwords, and compute term frequency counts using native PySpark DataFrame transformations.
CV: “Built a distributed log tokenization and frequency counter in PySpark avoiding collecting every row onto the driver.”
Incremental Spark MERGE, a simulated night job, dbt-shaped staging and tests, and a publish suite.
PySpark & Distributed Compute
PYSPARKOptimize slow SortMergeJoins between large fact tables and small dimension lookups by broadcasting the dimension to worker executors.
CV: “Optimized large-scale PySpark fact-to-dimension joins using BroadcastHashJoin to slash runtime by over 80%.”
PySpark & Distributed Compute
PYSPARKDeduplicate at-least-once distributed payment event streams by partitioning by merchant ID and ordering by timestamp with row_number().
CV: “Engineered deterministic streaming deduplication pipelines in PySpark using window ranking functions to eliminate duplicate ledger settlements.”
PySpark & Distributed Compute
PYSPARKDeduplicate streaming micro-batch events and retain the latest record per business key using PySpark window partitions.
CV: “Built PySpark streaming micro-batch deduplication pipelines using window ranking, eliminating 18,000 daily duplicate delivery records.”
PySpark & Distributed Compute
PYSPARKValidate raw IoT sensor bronze data, quarantine corrupted records with dead-letter routing, and promote clean silver telematics.
CV: “Designed PySpark lakehouse medallion ingestion pipelines with dual-routing dead-letter quarantine auditing for 12,000 IoT devices.”
PySpark & Distributed Compute
PYSPARKParse deeply nested semi-structured JSON payloads with StructType schemas, explode item arrays, and extract customer metadata.
CV: “Engineered high-throughput PySpark semi-structured JSON flattening pipelines parsing 3M daily orders with zero serialization overhead.”
PySpark & Distributed Compute
PYSPARKExecute idempotent upsert merges on hospital patient records in PySpark, updating existing records and inserting new profiles.
CV: “Designed idempotent CDC upsert merge synchronization pipelines in PySpark for 48 regional hospital registration databases.”
Late messages on a fake topic, a GCP free-tier deploy checklist, and the ship runbook order.
PySpark & Distributed Compute
PYSPARKEliminate distributed join stragglers and executor OOMs by splitting heavily skewed keys across parallel partitions using random salting.
CV: “Engineered distributed two-stage key salting pipelines in PySpark eliminating task stragglers and cutting execution time by 90%.”
Bigger projects you build over a few weeks on your own laptop. Each one comes with a week plan, starter code, viva questions with model answers and a resume line, so you can present it with confidence.
Intermediate · 5 weeks
Turn a raw sales file into a star schema and a dashboard a manager can read.
Open the planBeginner friendly · 4 weeks
Pull data from a public API every day, load it safely, and chart it.
Open the planBeginner friendly · 3 weeks
Clean messy student records and show placement rates by branch, skill and company.
Open the planBeginner friendly · 3 weeks
Check every new file against rules and send a clear pass or fail report.
Open the planIntermediate · 4 weeks
Spot unusual payments with SQL window functions and show them on a simple report.
Open the planIntermediate · 5 weeks
Turn a stream of page-view events into sessions, funnels and a daily report.
Open the planSame grain fights, in the studio.
The 12 to 18 month path that sequences these is on the roadmap.
