DevOps and quality
Terraform, Helm & Infrastructure as Code for Data Platforms
Describe data infrastructure as code with Terraform and Helm, and change it safely.
- Handwritten-style
- Diagrams and tables
- Worked examples
Who this notebook is for
Data engineers who now own infrastructure and need to describe a data platform as code and change it without breaking it.
What you will be able to do
- Explain how Terraform plans and applies changes
- Manage state safely, including locking and recovery
- Describe a data platform on AWS, GCP or Azure as modules
- Deploy Databricks, Snowflake and Kubernetes workloads from code
What's inside
Pick a chapter and the notebook opens right there.
- IaC Mental Model, Lifecycle & OpenTofuDeclarative state machines vs imperative scripts, plan/apply lifecycle, providers, state graph, Terraform vs OpenTofu vs Pulumi vs CDK, target scoping & non-deterministic plan mitigationStart here
- HCL Fundamentals, Dynamic Blocks & TypesResources, data sources, variables, outputs, locals, dynamic blocks, structural typing (object/tuple/map/set), count vs for_each, splats & built-in HCL functionsStart here
- State Management, Locking & State SurgeryRemote backends (S3+DynamoDB, GCS, Azure Blob), state locking deadlocks, state list/mv/rm, import {} blocks, moved {} refactors & automated drift detectionStart here
- Modules, Project Layout & TerragruntRoot vs reusable child modules, semver tags, monorepo vs multi-repo, environment layering (dev/stage/prod), Terragrunt DRY architecture & blast radius isolationStart here
- Data Platform on AWS: S3, Glue, MSK & EMREncrypted S3 lake buckets, KMS policies, Glue catalog/crawlers, Athena workgroups, Amazon MSK broker clusters, EMR on EKS & Lake Formation hybrid grantsStart here
- Data Platform on GCP & AzureGCS buckets, Pub/Sub topics, BigQuery datasets & authorized views, Cloud Composer (Airflow), Dataflow templates, ADLS Gen2, Synapse/Fabric & Workload Identity FederationStart here
- Databricks & Snowflake Providers as CodeSnowflake warehouses, databases, schemas, functional vs access roles, RBAC grants as code, Databricks Unity Catalog metastores, storage credentials & serverless warehousesStart here
- Kubernetes & Helm for Data WorkloadsNamespaces, RBAC, helm_release resource, values YAML layering, Spark-on-K8s Operator, Apache Airflow official Helm chart, Argo CD GitOps synchronization & SOPS encryptionStart here
- CI/CD for Infrastructure: OPA & InfracostGitHub Actions, Atlantis plan-in-PR workflows, Open Policy Agent (OPA/Rego), Sentinel, Infracost budget diffs, cloud OIDC passwordless auth & failed apply rollbacksStart here
- Security, Secrets & State EncryptionHashiCorp Vault dynamic credentials, AWS Secrets Manager, nonsensitive() hazards, native Terraform 1.7+ state encryption at rest, KMS rotation & SOC2/HIPAA complianceStart here
- Testing, Refactoring & Provider Upgradesterraform validate, tflint, native terraform test framework, Terratest in Go, blue-green infra, create_before_destroy, AWS v4-to-v5 upgrade migration & moved {} refactorsStart here
- Staff Drills & Master Cheat SheetMulti-account landing zone architecture, 200-resource DR rebuild, 50MB state corruption repair, FinOps GPU cost guardrails, CLI & HCL syntax cheat sheet & production checklistStart here
Practice this
Put it to work on LakeBench.
Common questions
- Does it cover OpenTofu?
- Yes. The first chapter explains the infrastructure as code model, the lifecycle and where OpenTofu fits.
- Is Helm included?
- Yes. A chapter covers Kubernetes and Helm for data workloads.
- Can I read a sample before I commit?
- Yes. The contents pages and the first pages of every notebook can be read online, so you can judge the style before going further.
