Day 16Phase 03 · Data Engineering Fundamentals

Files, Parquet and partitions

The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.

By the end of today you can

  • Explain why columnar formats like Parquet outperform row-based CSV for analytical queries.
  • Design Hive partition folder hierarchies that align with frequent query filter predicates.
  • Demonstrate partition pruning in query EXPLAIN plans and prevent small-file sprawl.

Why File Formats Matter: The 100 GB CSV Problem

ShopKart stores 100 GB of daily checkout data as CSV on cloud storage, with 60 columns per row. Finance wants revenue for 4 September.

Only 1 column is summed, but a CSV engine downloads and parses all 60.

This is the first section of the lesson. The rest unlocks with a pass.

Core topics: lake vs warehouse vs lakehouse, Medallion architecture, Parquet vs CSV, Hive partitioning, partition pruning, small files compaction

Build: Medallion Data Lake engine: raw CSV → bronze standardization → silver partitioned Parquet → gold daily sales aggregate.

Never destroy raw data. Store columnar Parquet with coarse temporal partitions so engines skip 90%+ of files.

Already have a pass? Go to your account. Need a pass? See pricing.