Foundations
Fundamentals of Data Engineering
What data engineers build and why: storage, pipelines, batch, streaming and the lifecycle.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Students, career switchers and analysts who want the whole data engineering picture before choosing which tool to learn next.
What you will be able to do
- Describe the data engineering lifecycle from source to dashboard
- Tell a data lake, a warehouse and a lakehouse apart and say when to use each
- Explain batch versus streaming, CDC and the medallion layout
- Name what makes a pipeline reliable, observable and cost-aware
What's inside
Pick a chapter and the notebook opens right there.
- The Big Picture & DE LifecycleWater Grid analogy, 5 lifecycle stages, schema-on-write vs readStart here
- Storage Systems & File FormatsBlock vs Object, Parquet row groups, Snappy/ZSTD, Avro & ProtobufStart here
- The Lakehouse & Table FormatsWarehouse vs Lake vs Lakehouse, Iceberg metadata tree, Delta Lake & HudiStart here
- Analytical Warehouses & Query EnginesOLTP vs OLAP, Snowflake/BigQuery decoupled compute, DuckDB & TrinoStart here
- Dimensional Modeling: Facts & DimensionsKimball vs Inmon, Star vs Snowflake, Surrogate keys & OBTStart here
- Advanced Dimensional Modeling: SCD & FactsSCD Types 1, 2, 3, 4, 6, fact table patterns & late arrivalsStart here
- Ingestion & Change Data Capture (CDC)Push vs pull, log-based CDC (WAL/Binlog), Debezium & Reverse ETLStart here
- Transformations & Modern ELTETL vs ELT, dbt models, materializations, incremental strategies & semantic layerStart here
- The Medallion ArchitectureBronze append, Silver cleanse, Gold marts, WAP pattern & GDPR purgeStart here
- Distributed Systems FundamentalsFallacies of computing, CAP theorem, hash partitioning & Raft consensusStart here
- Apache Spark: Architecture & ExecutionDriver, Executors, RDD vs DataFrame, DAG stages & shuffle boundariesStart here
- Spark Performance Tuning & JoinsCatalyst, Tungsten, Broadcast vs Sort-Merge joins, AQE & skew saltingStart here
- Event-Driven Systems & Apache KafkaCommit log model, topics, partitions, producers (acks/linger) & consumer lagStart here
- Advanced Streaming & Real-TimeISR replication, Exactly-Once (EOS), Event time, watermarks, windows & FlinkStart here
- Workflow Orchestration & AirflowDAG dependencies, Celery/K8s executors, Taskflow API (@task) & DagsterStart here
- Pipeline Reliability & OperationsIdempotency golden rule, backfilling, atomic swaps, retries with jitter & SLAsStart here
- Data Quality & Observability5 pillars of observability, Great Expectations, Soda, data contracts & OpenLineageStart here
- Governance, Security & FinOpsGDPR/PII masking, RBAC vs ABAC, row-level security & Snowflake/BigQuery cost cutsStart here
- Production Design PatternsLambda vs Kappa vs Lakehouse, Data Mesh, 100M clickstream blueprint & IoT pipelinesStart here
- FAANG Interviews & Senior Mindset4-step system design framework, ledger, ride tracking, Spark debug & 10 rulesStart here
- Modern Architecture & Open GovernanceData contracts (ODCS), open REST catalogs (Polaris), OpenLineage & FinOpsStart here
- Gap-Fill SupplementCapacity-planning math, streaming joins, OLAP cubes, sharding & replication, Snowflake essentialsStart here
Practice this
Put it to work on LakeBench.
Read the guide first
Short explainers that pair with this notebook.
- OLTP vs OLAP explained for beginners, with SQL examples
- Data warehouse vs data lake vs lakehouse: the differences, explained simply
- Medallion architecture (bronze, silver, gold) explained with a runnable example
- What is a data pipeline? A beginner's guide with a tiny example in SQL
- What is ETL? ETL vs ELT explained for beginners, with a runnable example
- Batch vs stream processing explained for beginners, with a SQL example
- CSV vs JSON vs Parquet: which file format should you use?
- How to become a data engineer as a fresher: a step-by-step roadmap
- Data engineer vs data analyst vs data scientist: the differences, explained simply
- Data engineering projects for beginners: 6 projects to build, in order
Common questions
- Is this the right notebook to start with?
- Yes if you want the big picture first. Read it alongside the Python and SQL notebooks, which teach the languages you will use day to day.
- Does it teach one specific tool?
- It introduces many tools at overview depth, such as Spark, Kafka and Airflow. For depth on any one of them, use that tool's own notebook.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
