Day 27Phase 05 · Spark / PySpark
PySpark Production Pipeline
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Enforce strict schema contracts at ingestion boundaries to prevent silent type corruption.
- Audit join cardinality pre- and post-join to detect accidental Cartesian row explosions.
- Write partitioned outputs idempotently with overwrite modes and circuit breaker assertions.
The 5-Stage Curation Pipeline
In a modern lakehouse, raw data arriving from upstream apps and payment providers is messy. Order amounts are negative and timestamps arrive in multiple formats.
Bronze preserves raw history for auditing, while Silver provides trusted relational data for downstream analytics models.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: data contracts & early validation, string & enum sanitation, windowed latest-record deduplication, join cardinality diagnostics, pre-write circuit breakers, idempotent writes
Build: End-to-end PySpark pipeline: raw ingestion → contract validation → window deduplication → broadcast join → pre-write assertions.
Verify dimension uniqueness before joining to prevent row explosion, and write with mode('overwrite') to ensure rerun safety.
Already have a pass? Go to your account. Need a pass? See pricing.