Overview
Least privilege, encryption at rest, and secrets that never land in git. IAM is the badge system for every API call.
On this page8 sections
The idea
A password committed to git
Imagine you paste a database password into a Python script so the nightly job can log in. Then you commit that file to git.
Everything works for a month. Then a teammate makes the repository public by mistake. Anyone on the internet can now read the password.
- Password pasted into the script
- Script committed to git
- Repo made public, password exposed
Deleting the line in the next commit does not help. Git keeps every old version, so the password is still in the history.
Never put a password in code. Git remembers everything.
Words you will see in this lesson
Six words. Each one has its own section below.
| Word | What it means |
|---|---|
| Secret | A password, key or token that proves who you are |
| Least privilege | Give a job only the access it needs, and nothing more |
| Service account | A login made for a program, not for a person |
| Encryption | Scrambling data so it is unreadable without the right key |
| Bucket | A folder in cloud storage |
| Secret manager | A locked vault in the cloud that hands passwords to programs that are allowed to ask |
Three habits for every pipeline
Three habits. If you remember nothing else, remember these.
| Habit | In plain words | What it stops |
|---|---|---|
| Least privilege | A job can touch only what it needs | A buggy script deleting the wrong data |
| Encryption | Data is scrambled on disk and while it travels | A stolen disk or a snooper on the network being able to read it |
| No secrets in git | Passwords are loaded when the job runs | A leaked password in the code history |
A pipeline gathers orders, emails and payment history into one place. One mistake can expose all of it at once.
Why this exists
In the cloud, a password is the front door
In an office, the building has locks and the network has a firewall. Only people in the building can reach the server.
In the cloud, anyone with a valid login can reach your data from anywhere in the world.
- Someone on the internet
- Shows a valid login
- Gets in, from any country
So your own rules matter most: who has a login, what each login can touch, and where the passwords are kept.
You and the provider each guard a part
The provider guards the buildings and the hardware. You guard everything you put inside your account.
This split is called the shared responsibility model.
| Part | Who looks after it | What that means |
|---|---|---|
| Buildings, power, servers | The provider (AWS, GCP, Azure) | Locks, guards, hardware security |
| Who has access | You | Give each role the smallest permission that works |
| Whether a bucket is public | You | Block public access |
| Passwords and keys | You | Keep them in a secret manager, not in code |
The provider secures the hardware. You secure what you put on it.
Most leaks are small mistakes
Big movie-style attacks are rare. Most leaks come from small, routine slips.
Three slips that cause most accidents.
| Mistake | Why it happens | The fix |
|---|---|---|
| Public bucket | Left open while testing | Turn on the account-wide block on public access |
| Admin rights for a script | Gave full access to make an error go away | Allow only the folders the script needs |
| Key file on a laptop | Downloaded for convenience, then committed | Let the cloud give the job a temporary key instead |
Picture this
One secure request
Here is a nightly job reading raw files and writing clean tables, with all three habits in place.
Read it left to right. First the permission check. Then the file is read from encrypted disk. The database password comes from the secret manager, never from the code.
Habit 1: give each job its own login
Run pipelines as a service account, not as your own person login. It is called an IAM role on AWS, a managed identity on Azure and a service account on Google Cloud.
Start with no access and add only what the job needs, such as read access to one folder. Do not give admin rights just to make an error go away.
- The machine starts
- It asks the cloud for a key
- It gets a key valid for about an hour
- The key expires on its own
Cloud machines can get a short-lived key like this automatically. If it leaks, it stops working soon, so the damage stays small.
Habit 2: scramble the data
Encryption scrambles data so that it looks like noise without the right key. There are two places it matters.
Both are normally switched on by default for cloud storage.
| Where | Name | What it protects against |
|---|---|---|
| On the disk | Encryption at rest | Someone stealing a disk or a backup |
| On the wire | Encryption in transit (HTTPS) | Someone listening on the network |
Some companies want to hold the key themselves. Each provider has a key service for this: AWS KMS, Azure Key Vault and Google Cloud KMS. If a job loses permission to use the key, it can no longer read the file.
Encryption does not stop someone who already has permission. That is why habit 1 comes first.
Habit 3: keep passwords out of git
If a password has ever been in a commit, treat it as public. Even five minutes is enough for a bot to copy it.
Git keeps the old version. If a secret was committed, create a new password, switch the job to it, and disable the old one. Cleaning the git history is a second step, not the fix.
- Commit 1: password added
- Commit 2: password deleted
- git log still shows the password in commit 1
If you ever find a leaked password, act in this order. First make a new password. Then switch the job to it. Then disable the old one. Only after that, worry about the git history.
The order matters, because the old password keeps working until you disable it.
Production jobs fetch passwords when they run, from a secret manager: AWS Secrets Manager, Azure Key Vault or Google Cloud Secret Manager.
Showing less of the table
Bucket rules decide who can open a whole file. For tables, warehouses can also hide single columns or single rows from certain people.
One table, three different views.
| Who is asking | Which rows they see | What is hidden |
|---|---|---|
| Finance manager | All orders | Card numbers are cut to the last 4 digits |
| Support engineer | Orders from their region | Card numbers show as XXXX-1234 |
| External auditor | Totals per country | Customer names are hidden |
This lets many teams use the same table without copying the data into separate versions.
A small example
Getting a password from the environment
The safe code only knows the name of the secret. The value is supplied when the job runs.
# wrong: the secret is part of the code, so it is part of git
db_password = "EXAMPLE_ONLY_DO_NOT_USE"
# right: the code only knows the NAME of the secret
import os
db_password = os.environ["DB_PASSWORD"]If DB_PASSWORD is missing, the second version stops right away with a KeyError. That is good, because it fails loudly. Never print the value to the logs.
On your laptop, keep local passwords in a file that is listed in .gitignore, so git never picks it up.
A least-privilege check in Python
A least-privilege rule is a list of allowed folders. Anything not on the list is refused.
Predict: which of these three paths will be denied?
ALLOWED_PREFIXES = ["bronze/orders/", "silver/orders/"]
def may_read(path, allowed=ALLOWED_PREFIXES):
return any(path.startswith(prefix) for prefix in allowed)
for path in [
"bronze/orders/dt=2026-08-26/part.json",
"bronze/payroll/2026-08.csv",
"silver/orders/summary.json",
]:
print(path, "->", "allow" if may_read(path) else "deny")bronze/orders/dt=2026-08-26/part.json -> allow bronze/payroll/2026-08.csv -> deny silver/orders/summary.json -> allow
The payroll file is denied because its path does not start with an allowed folder. The default answer is no.
Five questions before going live
Before a new pipeline goes to production, a reviewer asks five questions. Each one matches a habit.
HABITS = ["least-privilege", "encrypt", "no-secrets-in-git"]
print("every cloud job needs")
for habit in HABITS:
print("-", habit)every cloud job needs - least-privilege - encrypt - no-secrets-in-git
Five questions a reviewer asks about a new pipeline.
| Question | Habit | A good answer |
|---|---|---|
| Which identity does the job run as? | Least privilege | Its own service account, not a person's login |
| Which folders may that identity read? | Least privilege | orders/ only, not the whole bucket |
| Is the bucket encrypted at rest? | Encrypt | Yes, and we know who holds the key |
| Where does the database password come from? | No secrets in git | A secret manager or environment variable |
| Could a debug log line contain a customer's data? | No secrets in git | No, sensitive fields are masked before anything is logged |
Run through these five questions before every new pipeline. If you cannot say where a password comes from or which folders an identity can read, fix that first.
Watch the secured night job
The nightly worker signs in as its own service account with a temporary key.
- Scheduler
- Service account
IAM allows reading orders/ and refuses payroll/.
- IAM check
- Least privilege
The bucket decrypts the file for a job that is allowed to read it.
- Storage bucket
- Key service
The worker gets the database password from the secret manager, not from git.
- Secret manager
- Warehouse write
Four things to remember:
- Give each job its own login and only the folders it needs.
- Never commit a password. Load it when the job runs.
- Keep encryption on, at rest and in transit.
- Ask the five review questions before going live.
Common beginner questions
Why can't I use my own login for the job?
Your login stops working when you change your password or leave the company, and it probably has far more access than the job needs. A job should have its own identity, so you can limit it and see what it did.
If the provider encrypts everything by default, why does this matter?
Encryption protects against stolen disks. It does nothing against someone who holds valid permissions, or against a public bucket. That is why least privilege comes first.
Is an environment variable really safe?
It is much better than putting the password in code, and fine for learning. In production a secret manager adds access rules, a log of who asked and easy password changes. Start with environment variables and move later.
What comes next
Next, Why the cloud for data explains why companies move their data platforms to the cloud. After that, IAM goes through real permission files line by line.
Practice
Run Sample to see the habits printed. In the exercise, you get four security findings. Write risky() to pick the ones that break a rule: too much power, a secret in code, or no encryption. Store the ids in result and print them.
Going further: scan the settings, do not trust them
Most cloud data leaks are not clever attacks. They are settings left open: a public bucket, a wildcard permission, a key committed to code. These can be found by a simple scan of the configuration, which is why teams automate the check.
Below, a toy scanner reads a storage configuration as a dictionary and reports risky settings. Predict how many problems it finds in the first bucket.
def scan(bucket):
problems = []
if bucket.get("public_access"):
problems.append("bucket is public")
if not bucket.get("encrypted_at_rest"):
problems.append("not encrypted at rest")
if "*" in bucket.get("allowed_principals", []):
problems.append("anyone is allowed")
if not bucket.get("access_logging"):
problems.append("no access logging")
return problems
buckets = {
"raw-orders": {"public_access": True, "encrypted_at_rest": False, "allowed_principals": ["*"], "access_logging": False},
"curated-sales": {"public_access": False, "encrypted_at_rest": True, "allowed_principals": ["analytics-role"], "access_logging": True},
}
for name, config in buckets.items():
found = scan(config)
print(name, "->", found if found else "no problems found")raw-orders -> ['bucket is public', 'not encrypted at rest', 'anyone is allowed', 'no access logging'] curated-sales -> no problems found
The first bucket has 4 problems at once, which is typical of a bucket created quickly and never reviewed. The second passes every check. A passing scan does not prove safety, it proves that these specific mistakes are absent, so treat it as a floor, not a certificate.
| Risk | Why it happens | Fix |
|---|---|---|
| Public storage | A convenience setting during a test | Block public access at the account level |
| Wildcard permissions | Permissions that were too hard to work out | Grant specific actions on specific resources |
| Secrets in code | A key pasted in to make something work | Use a secrets manager and rotate keys that leaked |
| No logging | Not needed until something goes wrong | Turn on access logs before you need them |
Try a new scenario
You find an access key committed to a repository 2 weeks ago. The repository is private. What do you do, in order?
Work it out, then open the answer
Treat the key as compromised even though the repository is private: revoke or rotate it first, because anyone who had repository access, or any copy, could have used it. Then check the access logs for the last 2 weeks for use you do not recognise, remove the key from the history, and move the secret to a secrets manager. Deleting the file alone does not help, because the key remains in the history.
What is encryption at rest versus in transit?
At rest means data is encrypted while stored on disk. In transit means it is encrypted while moving over the network. You normally want both, and they protect against different problems.
Close the lesson and explain it
Without scrolling up, explain why a passing configuration scan is not proof of safety, and what you do when a key leaks.
Practicals · load into the editor
After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.
Finished this lesson?
Mark it as read when you could explain it to someone else.