Day 20Phase 04 · Cloud
Serverless query & catalogs (Athena, Synapse, BigQuery)
The curriculum for this day stays visible. The lesson, code, and workspace unlock with a pass.
By the end of today you can
- Explain the role of central metastore catalogs (AWS Glue, Synapse, BigLake) for serverless query engines.
- Create external tables over Parquet files and manage schema-on-read drift.
- Calculate serverless scan costs based on bytes read and optimize queries to minimize expense.
The Problem: Object Stores Do Not Understand SQL
On Day 19 you stored partitioned Parquet files in object storage. Object stores understand HTTP storage verbs: GET, PUT and DELETE, but nothing about tables, column types or SQL.
Central metastores (AWS Glue, Synapse, BigLake) register external schemas so serverless query engines (Athena, Synapse Serverless, BigQuery) can run interactive SQL directly over lake files.
This is the first section of the lesson. The rest unlocks with a pass.
Core topics: Storage + Catalog + Engine flow, Data catalogs & metastores, external tables & SerDes, serverless scan pricing, partition pruning, schema evolution
Build: Serverless query engine: catalog discovery → partition pruning benchmarks → pre-warehouse data quality validation.
Serverless query engines charge by data scanned. Pair Hive-partitioned Parquet with column pruning to cut 99% of costs.
Already have a pass? Go to your account. Need a pass? See pricing.