Day 19Phase 04 · Cloud
Object storage + data lake (S3, ADLS, GCS)
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Design bucket naming and key prefix architectures for bronze, silver, and gold lake tiers across S3, ADLS Gen2, and GCS.
- Apply lifecycle policies to transition cold historical data to cost-effective storage tiers.
- Demonstrate prefix-based partition pruning in analytical queries to minimize scan costs.
Object Storage: Flat Keys vs Hierarchical Namespaces
On a laptop, moving a folder of 50,000 files takes about a millisecond because the operating system only updates a pointer.
In flat object storage (S3 and GCS), there are no true folders; in ADLS Gen2, Hierarchical Namespaces provide atomic renames. Understanding how each cloud manages keys and namespaces determines how you design your data lake.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Object prefixes vs directories, Medallion zones (Raw/Silver/Gold), Hive partitioning, partition pruning, lifecycle tiering, compaction
Build: Object Storage Medallion Lake engine: raw ingestion → schema validation → Hive partitioned Parquet → curated sales mart.
Partition by query access patterns, keep immutable raw data for replay, and automate lifecycle tiering to cold storage.
Already have a pass? Go to your account. Need a pass? See pricing.