DevOps and quality
Data Quality, Testing & Observability
Tests, contracts and monitoring that catch bad data before your users do.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Engineers who are tired of finding bad data after the dashboard breaks and want a testing and monitoring approach for pipelines.
What you will be able to do
- Name the dimensions of data quality and the ways pipelines fail
- Place tests at the right level of a testing pyramid
- Use write-audit-publish and quarantine patterns to stop bad data early
- Set up reconciliation, anomaly checks and data contracts
What's inside
Pick a chapter and the notebook opens right there.
- Quality Dimensions, Failure Taxonomy & SLOsThe 6 core dimensions (completeness, accuracy, validity, timeliness, uniqueness, consistency), silent corruption, schema breakage, data SLAs/SLOs, error budgets & quality vs observabilityStart here
- Testing Pyramid for Data EngineeringUnit, integration, contract, schema & end-to-end tests, Chispa/pytest for PySpark transforms, property-based testing via Hypothesis, deterministic SQL test harnesses & slim CI lineage selectionStart here
- Great Expectations in DepthDataContext, expectation suites, checkpoints, actions, column/table expectations, Airflow integration, S3 Data Docs, scaling on 500M row tables & multi-batch partition delta validationsStart here
- Soda, dbt Tests, Deequ & PanderaTool selection framework by architectural layer, SodaCL declarative YAML checks, PyDeequ constraint verification on Spark, streaming Pandera schemas & historical metric trackingStart here
- Pipeline Patterns: WAP, Quarantine & BreakersWrite-Audit-Publish (WAP) pattern on Apache Iceberg, staging branches, quarantine error envelopes, automated circuit breakers & blocking vs non-blocking gate trade-offsStart here
- Automated Data ReconciliationRow count parity, cryptographic MD5/SHA-256 hash diffing, numeric checksums, CDC source-to-target reconciliation, penny-accurate financial balancing & SCD Type 2 timeline integrityStart here
- Statistical Anomaly Detection & ML DriftThe 4 observability pillars (freshness, volume, schema, distribution), rolling Z-scores, IQR, seasonality decomposition, KL divergence, PSI metric & holiday alert fatigue mitigationStart here
- Data Observability & Lineage ImpactLineage-driven blast radius analysis, OpenLineage standards, Marquez metadata harvesters, automated Data Diffing on pull requests & enterprise 2,000-table health scorecardsStart here
- Open Data Contracts (ODCS) & GovernanceThe Open Data Contract Standard, YAML schemas, producer-consumer SLA decoupling, CI breaking-change validation, Protobuf/Avro enforcement & organizational ownership modelsStart here
- Incident Management for Data PlatformsP0-P3 severity classification, containment runbooks, executive communication during revenue outages, blameless post-mortems (5 Whys) & idempotent partition backfill scriptsStart here
- Quality in Streaming & Real-Time MLOut-of-order event watermarks, sliding window null assertions, streaming Z-score monitors, training-serving skew detection, vector embedding health & continuous batch-stream parityStart here
- Staff Drills & Master Cheat Sheet5,000-table reliability rollout, $4.2M financial ledger silent corruption case study, zero-downtime contract migration across 40 consumers, master cheat sheet & production checklistStart here
Practice this
Put it to work on LakeBench.
Common questions
- Which tools does it cover?
- Great Expectations, Soda, dbt tests, Deequ and Pandera, plus open data contracts and lineage-based observability.
- Does it cover streaming data?
- Yes. A chapter covers quality in streaming and real-time machine learning pipelines.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
