Modern stack
Databricks Lakehouse for Data Engineers
Delta Lake, notebooks, jobs and the medallion layout on Databricks.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Engineers working on Databricks and candidates preparing for lakehouse interviews, from platform basics to production patterns.
What you will be able to do
- Explain Delta Lake internals and when to use liquid clustering
- Govern data with Unity Catalog and lineage
- Ingest at volume with Auto Loader and build Lakeflow declarative pipelines
- Deploy with Asset Bundles and control spend with system tables
What's inside
Pick a chapter and the notebook opens right there.
- Platform Architecture & Compute ModelsControl plane, data plane, account console, workspaces, classic compute vs serverless compute, editions & cross-account VPC securityStart here
- Delta Lake Internals & Liquid ClusteringTransaction log (_delta_log), JSON commits, Parquet checkpoints, OCC conflict resolution, Deletion Vectors, Liquid Clustering vs Z-order & CDFStart here
- Unity Catalog Governance & LineageMetastore, 3-level namespace (catalog.schema.table), storage credentials, external locations, volumes, automated column-level lineage & system tablesStart here
- Auto Loader & High-Volume IngestioncloudFiles source, directory listing vs file notification queues (SQS/Event Grid), schema inference & evolution, rescued data column & scalingStart here
- Lakeflow Declarative Pipelines (DLT)Declarative ETL, streaming tables vs materialized views, @dlt.expect quality rules, APPLY CHANGES INTO (CDC/SCD), and continuous vs triggered modesStart here
- Lakeflow Connect & Ingestion ArchitectureManaged connectors for Postgres, MySQL, Oracle & Salesforce, CDC log streaming, Kafka & Event Hubs, and COPY INTO vs Auto LoaderStart here
- Lakeflow Jobs & OrchestrationMulti-task DAGs, if/else branching, for-each dynamic loops, file-arrival & table triggers, repair-and-rerun workflows, parameters & error alertsStart here
- Compute, Photon Engine & PerformanceNative C++ vectorized Photon execution, Adaptive Query Execution (AQE), cluster sizing, Graviton/ARM CPUs, spot policies & predictive optimizationStart here
- Databricks SQL & Serverless WarehousesServerless SQL warehouses, auto-stop & auto- scaling, query federation (Snowflake, BigQuery, Postgres), Genie AI/BI spaces & multi-tier cachingStart here
- Structured Streaming on DatabricksMicro-batch vs continuous streaming, RocksDB state store provider, watermark drift, foreachBatch Delta MERGE INTO & end-to-end exactly-onceStart here
- Medallion Architecture in ProductionBronze append-only, Silver cleaned & conformed, Gold business aggregates, high-performance SCD Type 2 on 500M rows & domain data productsStart here
- Data Sharing & InteroperabilityDelta Sharing open protocol, Universal Format (UniForm) for Iceberg/Hudi, cross-engine querying from Snowflake/Trino & clean roomsStart here
- Security, Governance & ComplianceDynamic column masking, row filters, Attribute-Based Access Control (ABAC), PrivateLink network isolation & system.access.audit query log triageStart here
- DevOps, Asset Bundles (DABs) & CI/CDDeclarative bundle YAML, Git folders, multi- environment targets (dev/stage/prod), pytest & Databricks Terraform automationStart here
- Mosaic AI for Data EngineersFeature engineering in Unity Catalog, Vector Search delta sync indexes, MLflow model tracking & SQL AI functions (ai_query, ai_classify)Start here
- FinOps, System Tables & Cost ControlDBU pricing models, system.billing.usage queries, cluster policies, auto-termination governance, spot preemption & cost anomaly alertsStart here
- Migration Playbooks: Legacy to LakehouseHive Metastore (HMS) to Unity Catalog migration, Hadoop/EMR refactoring, Snowflake/Redshift to Delta cutover & dual-run validationStart here
- Staff Drills & Master Cheat SheetMulti-region global lakehouse, 50k events/sec CDC platform, $500k DBU runaway triage, 10-hour stalled ETL debugging & production checklistStart here
Practice this
Put it to work on LakeBench.
Read the guide first
Short explainers that pair with this notebook.
Common questions
- Is Spark knowledge required?
- It helps. If Spark is new to you, read the PySpark notebook first, then use this one for the Databricks platform itself.
- Does it cover migrations?
- Yes. A chapter gives migration playbooks from legacy systems to a lakehouse.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
