Cloud

Azure Cloud Data Engineer

ADLS, Data Factory, Synapse, Databricks on Azure and the interview questions around them.

  • Handwritten-style
  • Diagrams and tables
  • Worked examples
Start reading

Who this notebook is for

Engineers and candidates working on or interviewing for Azure data roles, including ADF and Synapse heavy teams.

What you will be able to do

  • Organise a lakehouse on ADLS Gen2
  • Build and parameterize pipelines in Azure Data Factory and Synapse
  • Place Databricks, Synapse and Fabric in the right part of the design
  • Ingest streams with Event Hubs and apply governance with Purview

What's inside

Pick a chapter and the notebook opens right there.

  1. Azure Data Lake Storage Gen2 (ADLS Gen2)Hierarchical Namespace (HNS), ABFS driver, Entra ID RBAC vs POSIX ACLs, tiers (Hot/Cool/Cold/Archive), WORM, Parquet & Delta layoutStart here
  2. Azure Synapse Analytics (Dedicated SQL Pools)MPP control/compute architecture, 60 distributed DBs, Hash/Round-Robin/Replicated distribution, CCI indexing, COPY statement, WLMStart here
  3. Azure Synapse Serverless SQL & Lake DatabasesDistributed Query Engine (DQE), OPENROWSET Delta/Parquet parsing, filepath()/filename() partition pruning, lake DBs & cost capsStart here
  4. Azure Data Factory (ADF) & Synapse PipelinesControl flow, Auto-Resolve vs Self-Hosted IR (SHIR), Mapping Data Flows, incremental CDC watermarking, retry patterns & CI/CD ARM templatesStart here
  5. Azure Databricks Architecture & Lakehouse EngineControl/Data plane, VNet injection, Photon engine, Unity Catalog 3-level namespace, Delta ACID log, Liquid Clustering & PySpark tuningStart here
  6. Azure Event Hubs & Real-Time IngestionPartitions, Consumer Groups, Throughput Units (TUs), Event Hubs Capture (Avro/Parquet to ADLS), Kafka wire compatibility & Geo-DR failoverStart here
  7. Azure Stream Analytics (ASA) & Real-Time ProcessingStreaming Units (SUs), Tumbling/Hopping/Sliding/Session windows, ASA SQL queries, watermarks, late arrivals & ASA vs Spark StreamingStart here
  8. Azure Cosmos DB for Data EngineersRequest Units (RUs), physical/logical partitioning, Change Feed pipelines, consistency levels, Synapse Link HTAP & NoSQL dimensional modelingStart here
  9. Microsoft Purview & Lakehouse Data GovernanceUnified Data Map, automated catalog scanning, classification rules, column-level data lineage, access governance & compliance auditsStart here
  10. Azure Lakehouse Security & Enterprise GovernancePrivate Endpoints, VNets, Azure Key Vault CMK, Dynamic Data Masking, RLS & multi-region LakehouseStart here
  11. Microsoft Fabric Architecture & OneLakeOneLake global virtualized data lake, Shortcuts, Delta Parquet V-Order, Direct Lake mode in Power BI & capacity sizingStart here
  12. Delta Live Tables (DLT) & Lakehouse PipelinesDeclarative ETL, @dlt.table streaming, Data Quality Expectations, Liquid Clustering & cluster autoscalingStart here
  13. Azure OpenAI, Cognitive Services & GenAI PipelinesEnterprise endpoints, PTUs, Azure AI Search HNSW vector storage, RAG pipelines & pgvectorStart here
  14. Azure Data Explorer (ADX / Kusto) & TelemetryColumnar extents, KQL query language, queued vs streaming ingestion, Materialized Views & time-series anomaliesStart here
  15. Azure FinOps & Cost OptimizationStorage lifecycle management, small file taxes, Dedicated Pool pause/resume, Databricks Spot fleets & zombie huntersStart here
  16. Hybrid & Multi-Cloud NetworkingExpressRoute, Virtual WAN, Private Link DNS Zones, Azure Storage Mover, GZRS replication & Data Mesh Hub-and-SpokeStart here
  17. Azure Production Observability, SRE & SLA MonitoringLog Analytics KQL, Event Grid dead- lettering, Event Hubs consumer lag & blameless RCAsStart here
  18. Staff Data Engineer FAANG Azure System Drills50M gamer telemetry on ADX, retail omnichannel Fabric, HPC risk simulation, IoT geofencing & $80k loop triageStart here
  19. Gap-Fill Supplement: Auto Loader, Fabric Mirroring & MessagingAuto Loader and Lakeflow on ADLS, Fabric Mirroring, Synapse Spark pools, Service Bus vs Event Hubs vs Event Grid, SQL MIStart here

Practice this

Put it to work on LakeBench.

Common questions

Does it cover Microsoft Fabric?
Yes. There is a chapter on Fabric architecture and OneLake, next to Synapse, Databricks and Delta Live Tables.
Is it only for Azure experts?
No. It explains each service from its purpose, so you can read it as an introduction or as a revision guide before an interview.
Can I read a sample before I commit?
Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.

More in Cloud

All notebooks