Overview
Least privilege, encryption at rest, and secrets that never land in git. IAM is the badge system for every API call.
On this page7 sections
The idea
Cloud security for a data engineer is three habits, not a certification. IAM (Identity and Access Management) decides who may do which action on which resource. Least privilege means the night job gets one prefix and one verb, not Owner on the whole project.
Encryption at rest means bytes on disk are encrypted so a stolen drive is not a stolen lake. The provider usually does this by default on object storage and warehouses. You still confirm it is on, and you still control who may decrypt by controlling IAM.
Secrets (access keys, tokens, database passwords) belong in a secret manager or in environment variables injected at run time. They never belong in source code, in a notebook you commit, or in a README screenshot. No IAM API runs in this tab. You name the habits in a Python list.
Why this exists
A public bronze bucket with customer emails is a breach. A service-account JSON committed to git is a credential anyone with repo access can use. A job running as a human user's keys means you cannot revoke the job without locking the person out.
Least privilege also limits blast radius when something is compromised. If the extract job can only read one vendor API and write one bronze prefix, a stolen key cannot DROP the gold dataset. If the job is Project Owner, it can.
Picture this
Think of a warehouse badge. People get badges. Robots (service accounts, instance roles, managed identities) get badges too. Each badge lists a verb (read, write, delete) and a door (this bucket prefix, this dataset). A master key is Action * on Resource *.
IAM is who and what. Encryption is the disk. Secrets stay out of the repo.
Humans log in with a user. Pipelines should run as a robot identity: a GCP service account, an AWS role the compute assumes, or an Azure managed identity. Sharing the reviewer's access keys with the scheduler is how keys leak into Git and chat logs.
A later IAM lesson parses a mock policy for the * on * statement. This lesson names the habits.
| Habit | Looks like in production | Beginner failure |
|---|---|---|
| Least privilege | storage.objects.get on gs://lake/bronze/orders/* | Action * on Resource * because onboarding was late |
| Encrypt at rest | Default bucket encryption, warehouse CMEK if required | Assuming 'it is in the cloud' means encrypted |
| No secrets in git | Secret Manager / env vars at run time | aws_secret_access_key in a .py file |
Encryption in transit is TLS on the API call (HTTPS). Encryption at rest is the disk. You need both. Turning off HTTPS 'just for the demo' is how tokens travel in clear text on a shared network.
A small example
List the three habits you will repeat on every job. The Python editor cannot attach a policy. It can grade that you remember the names.
HABITS = ["least-privilege", "encrypt", "no-secrets-in-git"]
print("every cloud job needs")
for habit in HABITS:
print("-", habit)A tiny policy sketch shows least privilege versus a master key. You are not calling IAM. You are reading the same JSON shape a later lesson will parse.
POLICY = {
"Statement": [
{"Sid": "ReadBronze", "Action": "storage.objects.get", "Resource": "gs://lake/bronze/*"},
{"Sid": "AdminEverywhere", "Action": "*", "Resource": "*"},
]
}
for stmt in POLICY["Statement"]:
wide = stmt["Action"] == "*" and stmt["Resource"] == "*"
print(stmt["Sid"], "MASTER KEY" if wide else "narrow")Common beginner questions
Is IAM only for people?
No. Most pipeline credentials are robots: service accounts, task roles, managed identities. Users are for humans in the console. Mixing them is how a departed employee's laptop still has production keys.
Does encryption mean I can share the bucket?
No. Encryption at rest stops a stolen disk. IAM still decides who may download the object. A public bucket with encryption on is still a public bucket.
Where do I put the password for now?
In this tab, you do not. In Core Python you used environment variables. In production you use AWS Secrets Manager, GCP Secret Manager, or Azure Key Vault, and IAM grants the job permission to read that one secret.
A green deploy with a key in git is still a fail
CI that prints secrets in logs is the same class of bug. Rotate anything that was committed. Add the file to .gitignore before the next push.
Config and secrets already taught os.environ
Do not print the token. Do not commit the token. IAM is the next layer: which robot is allowed to read the token from the secret store.
What comes next
The next module is Cloud Mental Model: which cloud this track walks through, and a Rosetta map of storage and warehouse names. After that, an IAM lesson parses a mock policy for the over-permissioned Sid.
Practice
Run Sample to print the three habits. Then complete Exercise: assign ["least-privilege", "encrypt", "no-secrets-in-git"] to result and print the list.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.