Lesson 2 of 4 in this module

2/14 in track

Cloud services overview

python12 min

Overview

Compute, storage, databases, networking, and IAM. Same jobs on AWS, GCP, and Azure, with different product names.

On this page8 sections

The idea

A menu with 200 items

Open the AWS, GCP (Google Cloud) or Azure console for the first time and you see more than 200 products. It feels like you must learn all of them.

  1. Full catalog, 200+ products
  2. Keep only what data work needs
  3. 5 kinds of product

You do not. Almost every data pipeline uses the same five kinds of product, and every provider sells all five.

Five jobs cover nearly every pipeline. A new product name is just one of those jobs with a brand on it.

Words you will see in this lesson

Six words. They come back all through the track.

WordWhat it means
ComputeA rented computer that runs your code
Object storageA place to keep files in the cloud, organised in buckets
DatabaseA program that stores tables
Network (VPC)A private space in the cloud where your machines can talk to each other
IAMThe rulebook for who is allowed to do what. Short for identity and access management
WarehouseA database built to add up millions of rows quickly, for reports

The five jobs

The five kinds of cloud product a data engineer uses.

JobWhat it doesIn a pipeline
ComputeRuns programs on rented CPU and memoryRuns the cleanup script
StorageKeeps files safely and cheaplyHolds raw CSV and JSON files
DatabaseStores tables you can query with SQLAnswers the analyst's questions
NetworkDecides which machines can reach whichKeeps the database off the open internet
IAMDecides who may do whatLets the script read one bucket and nothing else

The buttons and brand names differ between GCP, AWS and Azure. The five jobs do not.

Why this exists

Every team needs the same five things

Anyone building a data platform needs something to run code, somewhere to keep files, a database to query, a safe network and a set of access rules.

Providers (AWS, GCP and Azure alike) sell them separately, so you pay for each only while you use it. A script that runs 20 minutes does not need to pay for a database that sits idle all night.

  1. Timer starts the job
  2. Rent a small machine for 20 minutes
  1. Write the result to storage
  2. Machine deleted, bill stops
Watch outDo not memorise 200 names

When you meet a new product, ask which of the five jobs it does. Most of the time the answer is obvious from one line of its description.

Job posts use brand names for the same five jobs

One job post asks for Amazon S3 and Amazon Redshift. Another asks for Azure Blob Storage and Azure Synapse. A third asks for GCP Cloud Storage and BigQuery.

All three want the same thing: a place for files and a warehouse for reports.

Same job, three brand names.

Job post saysThe jobGoogle Cloud calls itAzure calls it
Amazon S3File storageCloud StorageBlob Storage
EC2ComputeCompute EngineVirtual Machines
Amazon RedshiftWarehouseBigQuerySynapse Analytics
AWS IAMAccess rulesCloud IAMEntra ID with RBAC

If you know one of them, you already know most of the other two. You only learn the new names.

Why the bill follows the job

Each job is billed in a different way. Compute is billed by the hour it runs. Storage is billed by the gigabyte per month. Network is billed by how much data leaves.

That is why a nightly job that runs 2 hours a day is cheap on rented compute, and why the same machine left on all month is not.

720 hours is 30 days of 24 hours. 60 hours is 30 nights of 2 hours.

PlanHours billed per monthRelative cost
Machine on all month720Highest
Machine on 2 hours a night60About one twelfth of the above

Picture this

One nightly job, five jobs working together

Diagram of a nightly job: compute reads from storage and loads a database, over a private network, with IAM checking each step
Five categories, one job. Under each name are the AWS, GCP and Azure products for that category.

Read it left to right. Compute reads raw files from storage and writes results into a database. The network is the private road between them, and IAM checks permission at every step.

One row per job, one column per provider.

JobAWSGoogle CloudAzure
ComputeEC2Compute EngineVirtual Machines
StorageS3Cloud StorageBlob Storage
DatabaseRDSCloud SQLAzure SQL
NetworkVPCVPCVNet
IAMIAMIAMEntra ID + RBAC

Three ways to rent compute

Compute comes in three shapes. Pick by how long the job runs.

Machine, container, function.

ShapeExamplesUse it forTime limit
Virtual machineEC2, Compute EngineLong jobs that need your own setupNone, runs until you stop it
Container serviceCloud Run, ECSThe same job packed so it runs anywhereDepends on the service
FunctionLambda, Cloud FunctionsSmall tasks, like reacting to a new fileAWS Lambda stops at 15 minutes

A function on GCP, AWS or Azure that has been idle takes a moment to start the first time it is called. That delay is called a cold start.

Row databases and column databases

Databases come in two kinds, depending on how they lay data out on disk.

Rows versus columns
Row store (for the app)Column store (for reports)On diskRecord 1: id, user, total, timeFast to save or fetch one orderSlow to add up one column over millionsof rowsAll the totals stored togetherFast to add up a column over millionsof rowsSlow to change one single record

A row store keeps each record together. A column store keeps each column together.

GCP Cloud SQL and Azure SQL are row stores, good for an app that saves one order at a time. BigQuery, Synapse and Redshift are column stores, good for reports.

Network and IAM both have to say yes

To reach a bucket, a machine needs a network route and permission. Missing either one fails, but the failure looks different.

A slow failure usually means network. A fast 403 usually means IAM.

Network routeIAM permissionWhat you see
MissingGrantedThe request hangs, then times out
OpenMissingAccess denied (403) straight away
OpenGrantedIt works

The network decides whether you can reach the door. IAM decides whether the door opens.

Who looks after what

Providers sell the same thing at three levels. The higher the level, the less you look after.

More provider work means less operations work for you.

LevelThe provider looks afterYou look afterExample
IaaS (infrastructure)Buildings and hardwareThe operating system, software and codeEC2, Compute Engine
PaaS (platform)Hardware and operating systemYour settings, jobs and tablesEMR, Dataproc
SaaS (software)Nearly everythingYour data and your queriesBigQuery, Snowflake

If you only need daily SQL reports, renting a bare machine and installing Spark yourself is a lot of extra work. A warehouse service gives you the result with far less to look after.

A small example

Looking up product names in Python

A dict inside a dict can hold the table above. The first key is the job and the second key is the provider.

Predict: what will the second line print?

PythonLook up compute names
SERVICES = {
    "compute": {"aws": "EC2", "gcp": "Compute Engine", "azure": "Virtual Machines"},
    "storage": {"aws": "S3", "gcp": "Cloud Storage", "azure": "Blob Storage"},
    "database": {"aws": "RDS", "gcp": "Cloud SQL", "azure": "Azure SQL"},
    "network": {"aws": "VPC", "gcp": "VPC", "azure": "VNet"},
    "iam": {"aws": "IAM", "gcp": "IAM", "azure": "Entra ID + RBAC"},
}

print(SERVICES["compute"]["gcp"])
print([SERVICES["compute"][cloud] for cloud in ("aws", "gcp", "azure")])
Compute Engine
['EC2', 'Compute Engine', 'Virtual Machines']

The first line looks up one name. The second line collects the compute product from all three providers.

From a product name back to its job

Sometimes you meet a name and need to know which job it does. The loop below checks each job's products until it finds a match.

Predict: what does it return for S3, VNet and Lambda?

PythonFrom product name back to job
def category_of(product):
    for category, names in SERVICES.items():
        if product in names.values():
            return category
    return None

print(category_of("S3"))
print(category_of("VNet"))
print(category_of("Lambda"))
storage
network
None

Lambda gives None because the small table has no function product in it. Lambda is compute, but it is not in our five-row table.

RememberTranslating a job post

When a post asks for S3, Blob Storage or Cloud Storage, read it as file storage. The brand changes, the job does not.

Counting what a team needs

A team usually runs several pipelines, and each needs a different mix of the five jobs.

PythonWhich job shows up most across a team's pipelines
PIPELINES = [
    {"name": "orders-nightly", "needs": ["compute", "storage", "database"]},
    {"name": "clickstream-stream", "needs": ["compute", "storage", "network"]},
    {"name": "finance-report", "needs": ["compute", "database", "iam"]},
]

def count_by_job(pipelines):
    counts = {}
    for pipeline in pipelines:
        for job in pipeline["needs"]:
            counts[job] = counts.get(job, 0) + 1
    return counts

print(count_by_job(PIPELINES))
{'compute': 3, 'storage': 2, 'database': 2, 'network': 1, 'iam': 1}

Compute is in all three pipelines, which is normal. Something always has to run the code.

counts.get(job, 0) means: use the saved count, or 0 if this job has not been seen yet. It avoids an error the first time a job appears.

Finding pipelines that use storage

PythonFilter pipelines by category
def needs_service(pipeline, category):
    return category in pipeline["needs"]

print("Pipelines needing storage:")
for p in PIPELINES:
    if needs_service(p, "storage"):
        print("-", p["name"])
Pipelines needing storage:
- orders-nightly
- clickstream-stream

A team asks this question before changing a storage rule: which pipelines will it affect?

Watch the five jobs work together

The five jobs in one run
  1. Raw events land in a bucket (AWS S3, Azure Blob Storage or GCP Cloud Storage).

    1. Raw events
    2. Object storage
  2. A timer starts a machine inside a private network (VPC).

    1. Timer
    2. Machine in VPC
  3. IAM checks that the machine's identity may read that bucket.

    1. IAM check
    2. Access granted
  4. The machine cleans the data and loads a table in the warehouse.

    1. Cleaning
    2. Warehouse table

Four things to remember:

  1. Every provider sells compute, storage, database, network and IAM.
  2. Pick compute by run time: functions (Lambda, GCP Cloud Functions) for short tasks, machines and containers for long ones.
  3. Row databases suit an app saving one order. Column databases suit reports over millions of rows.
  4. Reaching protected data needs both a network route and an IAM permission.

Common beginner questions

Which provider should I learn first?

The one you will actually use. If you have no preference, this track uses GCP for worked examples and names the AWS and Azure equivalent beside it.

Why are there more than five products then?

Each job has many flavours. Compute alone has machines, containers and functions. They all still sit under the same five headings.

Is IAM really something I rent?

Not in the usual sense, and it costs almost nothing. But every other service asks it for permission first, so it matters more than its price suggests. A later lesson goes deep on it.

What comes next

Next, Data lakes versus warehouses looks at where data lands: cheap raw files, or tidy SQL tables, and why teams use both.

Practice

Do the exercise with the SERVICES dict from the sample. A team lists 3 jobs and the kind of service each needs. Build result with the AWS product name for each job, in order, and print it.

Going further: keep compute near the data

Cloud services are billed in separate meters: storing, computing, and moving data. Moving data out of a region or out of the provider is often the surprise on the bill. The practical rule is to run compute in the same region as the data. The numbers below are illustrative, not a price list.

Assume moving data between regions costs 0.02 per GB. A job reads 5 TB a day. Predict the yearly cost of reading across regions, and compare it with moving the job to the data.

PythonThe cost of distance
GB_PER_TB = 1000
PRICE_PER_GB = 0.02            # illustrative number

daily_gb = 5 * GB_PER_TB
yearly_cost = daily_gb * PRICE_PER_GB * 365
print("read across regions:", round(yearly_cost), "per year")
print("read in the same region: transfer charge = 0")
read across regions: 36500 per year
read in the same region: transfer charge = 0

Reading 5 TB a day across regions costs about 36,500 a year in transfer alone, before any compute. Placing the job in the data's region removes that line. The same logic explains why a queue, a database and a function that talk constantly belong in one region.

MeterWhat you pay forHow to keep it small
StorageGB kept per monthCompress, expire old data, use cheaper tiers for cold data
ComputeMachine timeTurn it off when idle, size it to the job
Data transferGB leaving a region or the cloudKeep compute and data together, send results not raw data
RequestsNumber of reads and writesAvoid millions of tiny files

Try a new scenario

A dashboard in another region queries a warehouse for 200 GB of raw rows every hour. What two changes reduce the cost and the delay?

Work it out, then open the answer

Move the heavy work to the data: aggregate inside the warehouse so only a small summary crosses the region, and cache that summary so the dashboard does not repeat the query every hour when the data has not changed. Sending raw rows to be summarised far away pays for transfer and slows the dashboard at once.

What is serverless?

You run code or queries without managing machines. The provider starts capacity when needed and charges by use. Servers still exist, but you do not size, patch or keep them running.

Close the lesson and explain it

Without scrolling up, explain why data transfer between regions is an easy cost to miss, and how you avoid it.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Practice this

Same ideas as interview drills. These challenges open in the studio with a dataset and tests already set up.

Rate:
Was this useful?