System design interview

Cost Optimization for 50TB/Day Pipeline

HardLocked55 min read

Cost-optimize 50TB/day: Parquet/ZSTD, pruning, spot batch, autoscaling stream, hot/warm/cold lifecycle.

system-designcost-optimizationstoragestreamingbatch

Start here

The problem in plain words

A company collects 50 terabytes of data every day. Over a month that is about 1.5 petabytes, before any clever tricks. At cloud prices that single number becomes a budget line that gets you invited to uncomfortable meetings.

Cloud bills are made of meters. There is a meter for how much you store, one for how much data your queries read, and one for how many machine-hours you use. Cost work is mostly about turning each meter down without breaking the promises the business depends on.

A good answer does not say "use cheaper tools". It names the meters, picks the biggest levers for each, and protects the fast, critical paths while squeezing everything else.

Words you will meet

Compression
Making data smaller on disk. ZSTD is a compression method that shrinks columnar files a lot at modest CPU cost.
Parquet
A columnar file format. It stores columns together, compresses well and lets queries read only needed columns.
Partition pruning
Skipping data that cannot match a filter, so queries read less and cost less.
Retention
How long data is kept before it is deleted.
Hot, warm, cold storage
Tiers from fast and expensive to slow and cheap. Recent data stays hot, older data moves colder.
Lifecycle policy
An automatic rule that moves data between tiers and deletes it after a set time.
Spot / preemptible machines
Discounted cloud machines that can be taken back, good for work that can restart.
Autoscaling
Adding and removing machines automatically as the workload changes.
SLA
A promise about speed or freshness, such as a report ready by 6 AM.
Keeping everything, hot, forever
50 TBx 3050 TB a dayraw, uncompressedHot storagekept foreverCloud billgrows every month
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Every day another 50 TB lands in the most expensive kind of storage and nothing is ever deleted or moved.
  2. 2.That is about 1.5 PB more every month, and the bill rises steadily even though most old data is almost never read.

The numbers, as things you can picture

  • 50 TB a day, about 1.5 PB a monthBefore compression and lifecycle rules.
  • Parquet and ZSTD often make it 2.5 to 5 times smallerFrom 50 TB a day to roughly 10 to 20 TB, though this depends heavily on the data.
  • Raw hot 7 days, warm 60 days, cold 1 yearAn example lifecycle. Summaries are tiny, so keep them for years. Raw retention is what costs money.

What the interviewer wants

Show concrete ways to cut storage, scanning and compute cost at 50 TB a day, handling batch and streaming separately, while keeping the important deadlines.

The interview setup

Design a cost-optimized pipeline for 50TB/day. Discuss columnar formats (Parquet, ZSTD), partition pruning, spot for batch, autoscaling for streaming, and hot/warm/cold lifecycle.

Start by scaring the room with 50TB × 30 ≈ 1.5 PB/month before discipline. Then apply levers systematically.

Where cloud data bills come from

Stored bytes, scanned bytes (query engines), compute hours (batch/stream), and egress. Architecture maps to those meters.

Columnar + compression

Parquet with ZSTD (or similar) stores only needed columns for many queries and compresses well. Fat JSON as long-term truth is a cost anti-pattern.

Lifecycle tiers

Hot data on faster/expensive tiers for days. Warm compressed object storage for months. Cold/archive for years if compliance requires. Aggregates retained longer than raw.

Spot and autoscale

Batch on spot with restart safety. Streams autoscale on lag rather than "max peak forever."

Pruning

The cheapest byte is the one never scanned. Partitions and clustering enforce that.

The expanded problem

  • Estimate monthly storage under lifecycle policies
  • Optimize compute for batch and streaming separately
  • Ensure query pruning still works
  • Keep critical SLAs while cutting waste
Ingest --> hot (days) --> warm compressed --> cold archive
              \-> aggregates retained longer (small)

What good looks like

  • PB math first
  • Concrete levers with numbers
  • Separate batch vs stream compute strategy
  • Compliance exceptions called out
  • SLA-critical versus best-effort paths

Clarifying questions

  • Retention by dataset?
  • Which pipelines are SLA-critical versus best-effort?
  • Query patterns that must stay fast?
  • Any regulated "keep raw N years" requirements?

Out of scope

  • Negotiating vendor discounts as the whole answer
  • Deleting compliance data to hit a savings target

Clarifying questions strategy

Ask what changes grain, SLA, retention, or cost. State assumptions when answers are vague.

Scope control

Protect the critical path. Park optional marts after the SLA landing succeeds.

Interview framing details

You are expected to teach while you design. Start from a concrete failure, define the terms, then put the algorithm or layout on the board. Keep the scope tight: one grain, one SLA, one publication method.

Ask only the clarifying questions that would change your diagram. Write assumptions when the interviewer asks you to decide. Call out out-of-scope work so you do not burn the clock on tooling trivia.

A strong close restates the critical path, the failure mode you fear most, and what you would ship in week one.