Day 16Phase 03 · Data Engineering Fundamentals
Files, Parquet and partitions
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Explain why columnar formats like Parquet outperform row-based CSV for analytical queries.
- Design Hive partition folder hierarchies that align with frequent query filter predicates.
- Demonstrate partition pruning in query EXPLAIN plans and prevent small-file sprawl.
Why File Formats Matter: The 100 GB CSV Problem
ShopKart stores 100 GB of daily checkout data as CSV on cloud storage, with 60 columns per row. Finance wants revenue for 4 September.
Only 1 column is summed, but a CSV engine downloads and parses all 60.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: lake vs warehouse vs lakehouse, Medallion architecture, Parquet vs CSV, Hive partitioning, partition pruning, small files compaction
Build: Medallion Data Lake engine: raw CSV → bronze standardization → silver partitioned Parquet → gold daily sales aggregate.
Never destroy raw data. Store columnar Parquet with coarse temporal partitions so engines skip 90%+ of files.
Already have a pass? Go to your account. Need a pass? See pricing.