DevOps and quality

Git, Docker & CI/CD for Data Engineers

The Git, container and CI/CD workflow a data team uses to ship pipelines safely.

  • Handwritten-style
  • Diagrams and tables
  • Worked examples
Start reading

Who this notebook is for

Data engineers who write good pipelines but want to ship them with version control, containers and automated checks like a software team.

What you will be able to do

  • Branch, rebase and recover from a bad commit without losing work
  • Package a pipeline in a Docker image and keep the image small and safe
  • Run a local data stack with Docker Compose
  • Automate tests and deploys with GitHub Actions

What's inside

Pick a chapter and the notebook opens right there.

  1. Git Core Architecture, Branching & Remote WorkflowsGit DAG internals (blobs, trees, commits, SHA-1 object store), The 3 Trees (Working Dir, Index, HEAD), branch pointers, fast-forward vs 3-way merges, GitFlow vs Trunk-based branching, git fetch vs pull --rebase, and.gitignore secrets/big data isolationStart here
  2. Advanced Git, Rebasing & Emergency Disaster RecoveryInteractive rebase (git rebase -i squash, fixup, reword), resolving pipeline merge conflicts, git rerere, git stash (-u) & cherry-pick, binary search bug hunting (git bisect), reflog time-machine recovery for lost commits, and pre-commit hooksStart here
  3. Docker Fundamentals & Container Architecture for DEVMs vs Containers (Linux namespaces & cgroups isolation), Docker daemon, containerd & runc lifecycle, image layers & OverlayFS copy-on-write, multi-stage Dockerfiles for Python/PySpark, CPU/RAM resource limits, OOMKilled exit 137 triage, and exec/logs/statsStart here
  4. Docker Networking, Volumes & Persistent Data StorageBind mounts vs named volumes vs tmpfs, persistent database state (PostgreSQL, MinIO, Kafka), network drivers (bridge, host, none), custom networks & container DNS discovery, runtime env/secrets security, and non- root USER hardeningStart here
  5. Docker Compose & Local Modern Data Stack OrchestrationCompose file anatomy (services, networks, volumes), local PostgreSQL warehouse with auto-seeding DDL, MinIO S3 object lakehouse, Kafka event streaming cluster with KRaft, healthchecks & depends_on condition, and local Apache Airflow orchestrationStart here
  6. CI/CD Automation & GitHub Actions for Data PipelinesCI/CD philosophy for data teams, automated quality gates (Ruff, Black, SQLFluff), unit testing pipeline logic with pytest & mock data fixtures, GitHub Actions workflow architecture (runners, steps, matrix builds), building/pushing GHCR images, and end-to-end GitOps deploymentStart here
  7. Advanced Git Workflows & Enterprise Monorepo ManagementGit monorepos in enterprise data platforms, sparse checkout & partial clones (--filter=blob:none), submodules & subtrees for shared schemas/DDL catalogs, advanced pre-push validation hooks, worktrees for concurrent hotfixes, and FAANG corrupted repository recoveryStart here
  8. Deep Dockerfile Optimization & Multi-Stage Build EngineeringLean wheel builders for PySpark & C-extensions, BuildKit cache mounts (--mount=type=cache) & secret mounts, distroless images, layer invalidation, and zombie reaping (tini, dumb-init)Start here
  9. Container Security, Hardening & Enterprise ComplianceRootless Docker & user namespace isolation (UID 10001), dropping Linux capabilities (cap-drop=ALL), Seccomp syscall filtering, Trivy/Grype vulnerability scanning gates in CI, and read-only root filesystemsStart here
  10. Advanced Multi-Container Orchestration & Compose PatternsDRY configs via YAML anchors (&, *) & extensions (x-), deterministic startup sequencing (service_completed_successfully for DB seeders), multi-environment cascades, and Fluentbit loggingStart here
  11. Enterprise GitHub Actions & Advanced CI/CD WorkflowsMatrix testing across Python (3.9– 3.12) & PySpark (3.3–3.5), reusable workflows (workflow_call), composite actions, self- hosted ephemeral cloud runners, high-performance dependency caching, and keyless Cloud OIDCStart here
  12. Continuous Deployment, GitOps & Infrastructure as CodeGitOps principles for data lakes & warehouses, automated semantic versioning & tags, multi-architecture image builds (Docker Buildx AMD64/ARM64), automated database migration gates (Flyway/Alembic), and zero- downtime rolloutsStart here
  13. Container Resource Profiling, Metrics & Cgroups v2Cgroups v2 accounting, memory high vs max limits, CPU CFS bandwidth shares, saturation metrics (docker stats, cAdvisor), exit code 137 OOM root-cause forensics, OverlayFS vs volume I/O latency, and bridge vs host network throughputStart here
  14. Disaster Recovery, Container Post-Mortems & Incident TriagePost-mortem forensics via docker commit/export, troubleshooting distroless containers with ephemeral debug sidecars, rescuing corrupted named volumes & WAL recovery, container DNS split triage, and PID exhaustion triageStart here
  15. Staff DE Version Control, Containerization & Systems PhilosophyData vs code separation (DVC, LakeFS), The Twelve-Factor App for Big Data, Top 10 Docker anti-patterns, Top 10 Git anti-patterns, Dev Containers (.devcontainer) for 60s onboarding, and The Staff DE DevOps ManifestoStart here
  16. Modern Staff Kubernetes & Container InfrastructureK8s Jobs/StatefulSets & PV/PVC, Spark-on-K8s & Flink Operators, Supply Chain Cosign/SBOM/Trivy, and SOCI lazy-pullingStart here
  17. Gap-Fill SupplementGitLab CI/Jenkins/Azure Pipelines, Helm/Kustomize/Spark Operator, SOPS & Sealed Secrets, Testcontainers & CI testingStart here

Practice this

Put it to work on LakeBench.

Common questions

Do I need to know Docker or Git already?
No. The notebook starts from how Git stores history and what a container is, then moves to workflows used on data teams.
Does it cover Kubernetes?
Only at the end, as a modern look at container infrastructure. The focus is Git, Docker, Compose and CI/CD.
Can I read a sample before I commit?
Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.

More in DevOps and quality

All notebooks