System design interview

Log Aggregation System (10,000 Servers)

HardLocked55 min read

Collect logs from 10k servers at ~250 GB/sec aggregate, support sub-second error search for 7 days, and archive for 1 year.

system-designloggingsearchstorageestimation

Start here

The problem in plain words

Every program keeps a diary. When it starts, fails or does something interesting, it writes a line: a log. A company with 10,000 servers produces an enormous pile of diary lines every second. When something breaks at 3 AM, an engineer wants to type "checkout errors in the last hour" and get an answer in about a second.

The obvious idea is to feed every line into a search engine and keep it for a year. Search engines build a big index so that finding is fast, and building that index for everything costs a fortune in machines and storage. At this scale it is like hiring a second company just to run your search.

So the design question is what to make searchable and what to merely keep. You make a small slice, the errors of the last week, quickly searchable. You keep everything else cheaply, in case you ever need it.

Words you will meet

Log line
One line a program writes about what it is doing, for example "payment failed for order 812 at 03:02:11".
Agent
A small program on each server that reads its logs and sends them onward. Fluent Bit and Filebeat are examples.
Index
A lookup structure built in advance so a search is fast, like the index at the back of a book. Building and storing it costs a lot.
Hot storage
Data you can search interactively and quickly. Expensive per gigabyte, so you keep only recent data here.
Cold storage / archive
Cheap, slow storage for everything else, usually compressed files in object storage such as Amazon S3.
Backpressure
What happens when the receiver cannot keep up. The sender must slow down, buffer, or throw some data away.
Structured log
A log line written as labelled fields (service, host, level, message) instead of a free sentence, so you can filter on fields.
Sampling
Keeping only a fraction of very repetitive lines, for example 1 in 10, to control volume.
Quota
A limit on how much each service may send, so one noisy service cannot crowd out the others.
Why "index everything forever" does not work
all logssearchslow10,000 serversall writing logsOne big searchindexeverything, 1 yearEngineersearching at 3 AM
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Every line from every server, every level of detail, goes into one search index.
  2. 2.The one thing an engineer needs in an emergency, recent errors, now competes with an index full of data nobody ever searches.
  3. 3.The index and its storage grow enormous. Costs explode and searches get slower as the pile grows.

The numbers, as things you can picture

  • About 250 GB a second in the promptAn extreme number on purpose. Say that you would check whether every server really sends debug logs, and that even 1 to 5 GB a second needs a hot and cold split.
  • 1 second for a typical search over 7 daysThis is the promise for the hot tier, and only for errors.
  • 1 year of keep-everythingRequired, but only needs to be cheap, not instant.
  • If errors are 1 line in 100Indexing only errors needs about 100 times less hot capacity. That single filter often decides whether the cluster is possible.

What the interviewer wants

Design collection from 10,000 servers, a fast search for recent errors, a cheap one-year archive, and rules that stop logging from ever slowing the real applications.

Interview framing

You are designing a log aggregation platform for a large fleet. Interview length about 55 minutes. The prompt will throw an extreme ingest number at you on purpose. Strong candidates negotiate the number, then separate hot searchable errors from cheap long-term archive.

The prompt:

Collect logs from 10,000 servers. Each server generates about 50,000 log lines per second in the aggressive version of the prompt, with aggregate volume around 250 GB/sec. You need full-text search on error logs with sub-second latency for the last 7 days, and archive all logs for 1 year.

Background from first principles

Logs are append-only records of what software did: info, warn, error, debug. Operators search them during incidents. Compliance may require long retention. Those two needs fight each other. Full-text indexes are expensive. Object storage is cheap. If you put every line for a year into Elasticsearch, you are designing a second company.

Agents on hosts (Fluent Bit, Vector, and similar) must protect the application. If the logging backend is down, the agent buffers then drops with metrics. Logging must never deadlock checkout.

Understand the access patterns before picking stores:

  • Incident response: recent errors, filter by service and time, sub-second.
  • Postmortem deep dive: maybe weeks ago, can be slower.
  • Compliance audit: prove retention, rarely interactive search.

Those are different systems wearing one product name.

Expanded problem and constraints

  • Fleet: 10k servers.
  • Prompt scale: about 250 GB/sec aggregate. Treat as interview extreme. Say if you negotiate down to a softer 1 to 5 GB/sec while keeping the same shape.
  • Hot path: error (and maybe warn) logs searchable for 7 days, typical queries under about 1 second.
  • Cold path: all levels archived 1 year with compression and lifecycle tiering.
  • Filters: service, host, severity, time range.
  • Agents: restarts should not create silent huge gaps. Detect gaps.
  • Cost: full-text indexing everything for a year is usually not viable. Say so.

What good looks like

You challenge 250 GB/sec politely. You route severity=error to a hot search cluster with index lifecycle management. You ship everything to object storage. You design agent backpressure and per-service quotas. You explain how to find logs from day 60 (cold scan or rehydrate), not by keeping a year of hot indexes.

You also mention structured JSON fields so filters do not depend on brittle free-text alone.

Clarifying questions

  • Is 250 GB/sec real or a stress number we can negotiate?
  • Must all log levels be searchable, or only error/warn?
  • Structured JSON logs vs free text?
  • Retention and compliance (PII or secrets in logs)?
  • Multi-region apps?
  • Who pays if one noisy service emits 80% of volume?
  • Are trace ids standard across services?

Scale and estimation prompts

  • Even at 5 GB/sec, 7 days of raw is huge; indexed copies multiply cost.
  • Errors are a small fraction of total volume. Index selectively.
  • 1 year archive belongs in object storage with gzip/zstd and cold tiers, not forever-hot search nodes.
  • Search QPS during incidents spikes; size hot cluster for query concurrency on the error subset.
  • Show the shape at negotiated scale and at prompt scale so the interviewer sees dials, not panic.

Work a softer estimate out loud: 10k hosts x 1 MB/s is 10 GB/s. Still enormous. Sampling debug and indexing only errors remains mandatory.

Out of scope for v1

  • Perfect fidelity debug search for every line for 365 days.
  • Blocking application threads when the logging sink is slow.
  • A single cluster that is both unlimited ingest and unlimited retention full-text.
  • Guaranteeing zero log loss under total backend outage (prefer drop with loud metrics).
  • Building a full APM traces product unless asked (logs vs traces vs metrics).