Modern stack
dbt for Data & Analytics Engineers
Models, tests, sources and docs in dbt, and how a project is structured.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Analysts and engineers adopting dbt, and candidates who need to explain how a dbt project is structured and tested.
What you will be able to do
- Choose the right materialization for a model
- Write incremental models and Jinja macros
- Track history with snapshots and protect quality with tests and contracts
- Set up CI with slim builds and document lineage
What's inside
Pick a chapter and the notebook opens right there.
- What dbt Is & Core LifecycleT in ELT, compile vs run lifecycle, AST generation, project layout, profiles.yml, dev/staging/prod targets, environment variables, dbt Core vs Cloud vs Fusion engineStart here
- Models & Materializationsview, table, ephemeral, incremental, materialized_view; ref() and source() DAG lineage building; node selection syntax (tag:, +, @, intersection & selectors.yml)Start here
- Incremental Models in Depthis_incremental() macro, schema evolution, strategies (append, merge, delete+insert, insert_overwrite, microbatch), unique keys, late data, lookbacks & full-refresh trapsStart here
- Jinja & Custom MacrosControl flow, context variables, custom DDL generators, run_query(), dynamic column pivots, packages (dbt_utils, dbt_expectations, codegen) & cross-cloud adapter.dispatch()Start here
- Sources, Seeds & Snapshots (SCD2)source freshness, max_loaded_at, warn/error thresholds; static seeds & limits; Snapshots SCD Type 2 (timestamp vs check strategies, dbt_valid_to & hard deletes)Start here
- Testing & Quality AssuranceGeneric tests (unique, not_null, accepted_values, relationships), singular tests, store_failures, limit/where clauses, unit tests (dbt 1.8+ mock fixtures) & layer test strategyStart here
- dbt Mesh, Contracts & GovernanceModel contracts (enforced: true), column data types & constraints, model versioning & deprecation, dbt Mesh (cross-project ref, access modifiers, groups)Start here
- Project Structure & Dimensional ModelingStaging (stg), Intermediate (int), Marts (fct, dim); grain discipline & surrogate primary keys; Kimball star schemas vs One Big Table (OBT) in columnar cloud warehousesStart here
- Semantic Layer & MetricFlowSemantic models, entities, measures, metric types (simple, cumulative, derived, ratio, conversion), dimensions, time grains, BI tool exposures & join fan-out preventionStart here
- Documentation & Metadata Lineagedocs generate, doc blocks, markdown overviews; metadata artifacts (manifest.json, run_results.json, catalog.json); column-level lineage & automated catalog synchronizationStart here
- Performance, Cost & Warehouse TuningSnowflake clustering, BigQuery partition/cluster, Delta liquid; threads tuning & parallelism; query tagging with dbt query_comment & slashing warehouse bills by 50%Start here
- CI/CD & Slim CI DeploymentSlim CI with state:modified+ and manifest deferral (--defer -- state); GitHub Actions pipelines; ephemeral PR schemas; orchestrator integration (Airflow Cosmos, Dagster & dbt Cloud)Start here
- Python Models & Alternative Enginesdbt Python models (Snowpark, PySpark, BigQuery DataFrames); ML feature preparation & NLP; comparative architecture: dbt Core vs SQLMesh virtual environmentsStart here
- Staff Drills & Cheat Sheet1,000-model enterprise platform architecture, silent metric drift incident postmortem, 50TB broken incremental recovery runbook, Senior AE interview deep- dive & Ultimate dbt Staff Cheat SheetStart here
Practice this
Put it to work on LakeBench.
Read the guide first
Short explainers that pair with this notebook.
Common questions
- Do I need to know SQL first?
- Yes. dbt is SQL plus structure, so basic SELECT, JOIN and GROUP BY knowledge helps. The SQL notebook covers that.
- Does it cover the semantic layer?
- Yes. There is a chapter on the semantic layer and MetricFlow, and another on dbt Mesh and contracts for larger teams.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
