Skip to content
LakeBench
ProblemsCommunityPricing
Sign inStart practicing
Back
  1. Home
  2. Interview prep
  3. Egress and hidden cloud costs

Cloud · Cloud Architecture Basics

Egress and hidden cloud costs

Easycloud-45
cloud-costsfinopsegressobject-storagecost-optimization

Question

Which cloud costs surprise data teams most?

Solution

Data engineering teams are most frequently surprised by network egress fees, object storage API request costs, idle compute clusters, and unmetered analytical query scans. Moving data across cloud regions, between availability zones, or over the internet generates massive network bills that often surpass raw storage costs. Organizations control these expenses by implementing granular resource labeling, setting automated budget alerts, compaction routines for small files, and query governance guardrails.

The primary drivers of unexpected cloud bills

While storage capacity fees are cheap and predictable, data movement and API interaction costs scale non-linearly with workload volume.

Raw Storage (1 TB Parquet)         ---> ~$20 / month
Network Egress (1 TB to Internet)  ---> ~$90 / month
Object Store PUT Calls (10M files) ---> ~$50 / batch (exceeds storage cost)

Engineers watch for these specific operational budget drains:

  • Network egress fees incur steep charges when moving data across cloud provider borders, between distinct regions, or across availability zones within the same region.
  • API request charges on object stores like Amazon S3 and Google Cloud Storage quickly overwhelm storage fees when pipelines produce millions of tiny files. Writing ten million small files costs fifty dollars in PUT requests on S3, far exceeding the pennies spent storing the data bytes.
  • Idle compute clusters left running overnight, such as forgotten development Dataproc or EMR clusters without auto-termination policies, burn thousands of dollars in unallocated compute hours.
  • Excessive logging volume generated by verbose Spark or Airflow tasks sent to Amazon CloudWatch or Google Cloud Logging can produce surprise monitoring bills.
  • On-demand query engines (such as running unpartitioned queries in BigQuery or Athena) scan entire terabyte tables for a single analyst report.

Implementing FinOps governance

Managing cloud spend requires systematic financial engineering practices:

  • Enforce mandatory resource labeling with tags for project, team, environment, and pipeline name to attribute costs directly to business units.
  • Configure cloud budget thresholds and programmatic billing alerts to notify engineers before monthly spending limits are breached.
  • Implement file compaction jobs to combine tiny files into optimal 128MB to 512MB chunks, shrinking API request volumes.
PreviousNext