Overview
Source volume, fan-in, Kafka versus files, and backpressure. Streaming already taught partitions and at-least-once. Here you size the front door.
On this page8 sections
The picture
Suppose a ShopKart flash sale makes orders arrive at 8 times the normal rate for 30 minutes. If the pipeline's front door, where raw events first enter, was sized for the average day, its buffer fills up and checkout has to wait for it.
The front door of a data pipeline must be sized for the busiest hour, not the average one.
This lesson sizes that door for 500 million events a day, using the workload brief established in [Requirements to numbers](/learn/system-design/design-req) and [Batch vs stream](/learn/system-design/design-batch-stream).
3 sources, one contract. The buffer is a queue or an object prefix if you stayed batch.
Fan-in means several sources, here checkout, a tracking pixel and a vendor feed, merging into one buffer without overwhelming whatever reads from it.
The streaming track taught topics, partitions, keys and at-least-once delivery. This lesson sizes the front door those lessons described. No broker runs in this tab.
Words you will see in this lesson
Six words. The rest of the lesson uses only these.
| Word | What it means |
|---|---|
| Ingest | Take data in from the source |
| Producer | Whatever sends events in, like an app |
| Partition key | The value that decides which lane an event goes to |
| Backpressure | When the receiver is slower than the sender |
| Sink | Where the data ends up |
| Schema | The agreed list of fields and their types |
Why this shape
The ingest layer decides how much can run in parallel, whether order is kept, and what happens when the sink is slow. Too few partitions stall at the peak. One hot key stalls one lane. Assuming the sink is infinitely fast invites an outage.
Bytes matter as much as counts. A 2 KB JSON event and a 200 byte binary event are very different loads at 23,000 events a second. Size for the peak hour, not the daily average.
Work it out for the flash sale. If the average day is 5,800 events a second and the sale multiplies that by 8, the door must absorb about 46,000 a second for half an hour.
A buffer sized for the average fills up in seconds and the checkout page times out.
Walk the boxes
Teaching arithmetic. Real peaks follow product launches. Size for the hour, not the average.
| Slice | Events | Rough rate |
|---|---|---|
| Day | 500M | ~5.8k/s |
| Peak hour (4x) | ~83M | ~23k/s |
| Quiet hour (0.3x) | ~6M | ~1.7k/s |
| 1 KB event, peak | ~23 MB/s inbound |
Files or a queue? Follow the source. If checkout publishes to a topic, meet it there. If a vendor drops gzip files in a bucket, list the folder. Turning a file into a stream, or a stream into a file, adds work nobody asked for.
Object prefixes for file-based sources. Topics for live service producers.
The number of partitions limits parallelism, and the key pins one entity to 1 partition so its events stay in order. Choose a key with very many distinct values, such as user_id or session_id, never one with a few values such as country.
Same topic, 3 partitions. Key = user_id so one person's clicks stay ordered.
A matching example
Size the topic for 23,000 events a second at the peak, choose user_id as the key, and decide what happens when the sink falls behind.
High-cardinality user_id spreads load. Country would hot-spot India.
| event_id | user_id | country | partition key |
|---|---|---|---|
| e-001 | u-8821 | IN | u-8821 |
| e-002 | u-4410 | US | u-4410 |
| e-003 | (none) | IN | UUID fallback |
Predict first: if 1 partition handles about 5,000 events a second, how many do you need at 23,000?
import math
def choose_partition_key(event):
"""Pick a high-cardinality key that distributes evenly."""
if "user_id" in event:
return event["user_id"]
# Bad key: country has ~200 values and creates hot partitions
return event.get("session_id", "fallback-uuid")
def estimate_partitions(peak_rate, target_per_partition=5000):
"""Size partition count for peak throughput, rounding up."""
return max(math.ceil(peak_rate / target_per_partition), 3)
print(estimate_partitions(23_000))
print(estimate_partitions(1_000))5 3
23,000 divided by 5,000 is 4.6, which rounds up to 5. Rounding down to 4 would leave you short at the peak, so always round up, and add headroom. The floor of 3 is a sensible minimum even for a quiet topic.
If a large share of traffic shares one partner id, that partition becomes the bottleneck. Salt the key or isolate that partner on its own topic.
What if one key is much bigger than the rest?
Say one big merchant generates 40% of all orders and you chose merchant_id as the key. Partition 3 drowns while the others sit idle.
Two fixes: add the order id to the key so the merchant's orders spread out, or add a small salt of 0 to 7 so one hot key becomes eight.
def salted_key(merchant_id, order_id, buckets=8):
return f"{merchant_id}_{sum(ord(c) for c in order_id) % buckets}"
orders = [f"ORD-{i}" for i in range(1, 9)]
print(sorted({salted_key("M-1", o) for o in orders}))['M-1_0', 'M-1_1', 'M-1_2', 'M-1_3', 'M-1_4', 'M-1_5', 'M-1_6', 'M-1_7']
The merchant now lands on eight different keys. The cost is that something downstream must add the eight pieces back together, usually with a 2-step aggregation.
What happens when the sink is slower than the producers?
That is backpressure. You have three choices. Block the producers, which is honest if they retry and the sink ignores repeats.
Drop events, which needs an explicit budget and an alert. Or spill to a dead-letter queue, which is a leak unless a job drains it.
- Block: producer retries; at-least-once will fire; idempotent sink required.
- Drop: only with an explicit budget and a page threshold.
- Spill / DLQ: quarantine, count, drain on a schedule.
# Backpressure strategy skeleton
class IngestBuffer:
def __init__(self, max_queue_size: int = 100_000):
self.max_queue_size = max_queue_size
self.dlq_count = 0
def accept(self, event: dict) -> str:
if self.queue_size() < self.max_queue_size:
self.enqueue(event)
return "accepted"
elif self.can_drop(event):
self.increment_drop_metric()
return "dropped" # only low-priority events
else:
self.send_to_dlq(event)
self.dlq_count += 1
if self.dlq_count > 10_000:
self.page_oncall("DLQ growing fast")
return "quarantined"The database, the warehouse, and the queue all have throughput limits. Size the sink write path for peak rate, not average. Test what happens when the sink is temporarily down.
Assume duplicates. Design the sink as if every event may arrive twice. Orchestration idempotent-load is how you survive that assumption.
You practiced exponential backoff in the Core Python track. The same pattern applies to ingest producers that get a 503 from the buffer. Retry with backoff, then page if retries exhaust.
Which format should events use on the wire?
At 500 million events a day the format changes the network bill. JSON is easy to read but large. Binary formats are smaller and enforce a schema.
Serialization format trade-offs at 500M daily event scale.
| Format | Average Payload Size | Schema Enforcement | CPU Deserialization Cost |
|---|---|---|---|
| JSON | 1,200 bytes | Implicit / Runtime validation | High (string parsing, unicode decoding) |
| Apache Avro | 220 bytes | Strict (Confluent Schema Registry) | Low (binary compact encoding) |
| Protocol Buffers | 180 bytes | Strict (compiled IDL definitions) | Very low (optimized native binary bindings) |
At 1,200 bytes, 500 million events are about 600 GB a day. At 180 bytes they are about 90 GB. A schema registry also stops a producer from shipping a breaking change to a live topic.
Protecting the door at the edge
| Technique | What it does | Plain example |
|---|---|---|
| Rate limit (token bucket) | Gives each client a quota and answers HTTP 429 beyond it | A buggy partner app cannot flood you |
| Circuit breaker | Past 90% queue capacity, sheds low-value telemetry and keeps purchases | Page views are dropped before payments are |
| Backoff with jitter | Clients wait base * 2**attempt plus a random amount between retries | Thousands of retrying phones do not arrive in a synchronized wave |
| Compression (LZ4, zstd) | Shrinks the bytes on the wire at a small CPU cost | LZ4 can cut JSON traffic by about 65% |
| Regional entry points | Accept traffic near the user, then ride the cloud's own network | A phone in Singapore does not cross the public internet to one region |
The backoff schedule is easy to see. With a base of 0.1 seconds and a cap of 5, successive retries wait 0.1, 0.2, 0.4, 0.8, 1.6, 3.2 and then 5 seconds, with a little randomness added each time.
Common beginner questions
What if I only have 1 million events per day, not 500 million?
The same principles apply at smaller scale, but the urgency is lower. At 1 million per day you can get away with simpler tooling (a single file dump, a small queue). The partition math just yields fewer partitions. Learn the pattern at 500M so you can scale down confidently.
Do I need Kafka specifically?
No. Kafka is one implementation of a distributed message queue. Google Pub/Sub, Amazon Kinesis, Azure Event Hubs, and Redpanda all fill the same architectural slot. The design brief cares about the pattern (queue with partitions and consumer groups), not the vendor.
What happens if the system gets 10x more data overnight?
If your partition key is high-cardinality, you add more partitions and more consumer instances. If you hardcoded a low-cardinality key (like country), you are stuck. Partition key choice is the decision that is hardest to change later.
Problem: A flash sale pushes orders to 8 times normal. A buffer sized for the average fills in seconds, and checkout times out.
- 8x spike
- buffer full
- checkout hangs
Rate: Start from the brief: 500M events a day is 5,800 a second on average and about 23,000 at the 4x peak. Size the door for the peak.
- 5.8k/s avg
- 23k/s peak
Door: Pick the door by the source: a topic for live services, an object folder for vendor files. Choose user_id as the key so one person's events stay in order.
- topic or files
- key = user_id
Sizing: estimate_partitions(23_000) is 4.6, rounded up to 5. Rounding down would leave the door short at the peak.
- 23,000 / 5,000
- round up
- 5
Hot key: One merchant with 40% of orders is a hot key. Salt it across eight lanes and add the pieces back downstream.
- M-1
- M-1_0..7
Backpressure: When the sink is slow, block with retries, drop with a budget or spill to a drained dead-letter queue. Design the sink for duplicates.
- block
- drop
- DLQ
What comes next
The next lesson covers storage layout: how to land those 500 million events into bronze, silver, and gold layers so that downstream jobs can prune to 1 day without scanning the entire lake.
Practice
Run estimate_partitions(23_000) and confirm the answer is 5. Then write one sentence: why country is a bad partition key at this volume.
Going deeper: sizing 500 million events a day from the requirements
A design for 500 million events a day should read like arithmetic that ends in decisions, not a list of technologies. This section does that arithmetic step by step, states every assumption, and shows which decision each number drives. Change an assumption and the numbers change; that is the point.
State the assumptions first
| Assumption | Value used here | Why this is a guess you should confirm |
|---|---|---|
| Events per day | 500,000,000 | Given |
| Average event size | 1 KB (JSON, uncompressed) | Measure real payloads; one large field changes everything |
| Peak to average ratio | 4x | Look at a real traffic curve; a sale or a push notification can be far higher |
| Kafka retention and replication | 7 days, 3 copies | Retention sets how far back you can replay; replicas buy durability |
| Consumer speed | 2,000 events/s per consumer instance | Benchmark your actual handler; this depends on the work done per event |
| Lake compression | Parquet about 5x smaller than raw JSON | Depends on the data; measure a sample |
Step 1: rate
A day has 86,400 seconds. Size for the peak, not the average.
| Quantity | Calculation | Result |
|---|---|---|
| Average events per second | 500,000,000 / 86,400 | about 5,787 per second |
| Peak events per second | 5,787 x 4 | about 23,148 per second |
| Average bandwidth | 5,787 x 1 KB | about 5.8 MB per second |
| Peak bandwidth | 23,148 x 1 KB | about 23.1 MB per second |
Step 2: storage
Retention and replication multiply the disk bill, so they are the first cost lever.
| Place | Calculation | Result |
|---|---|---|
| Raw data per day | 500,000,000 x 1 KB | 500 GB |
| Kafka on disk (7 days, 3 copies, before compression) | 500 GB x 7 x 3 | 10.5 TB |
| Kafka with 4x compression | 10.5 TB / 4 | about 2.6 TB |
| Lake (Parquet, 5x) | 500 GB / 5 | 100 GB per day, about 36.5 TB per year |
Step 3: parallelism
At 2,000 events per second per consumer, peak needs 23,148 / 2,000, about 11.6 consumers. Add 2x headroom for retries, deploys and a slow broker, and you get about 24.
Kafka cannot give more consumers than partitions work, so choose at least 24 partitions, and round up (for example 32) so you can grow without repartitioning.
Step 4: recovery
Suppose the consumers are down for 1 hour. At the average rate, 5,787 x 3,600 is about 20.8 million events waiting in the log. With 24 consumers handling 48,000 per second and 5,787 per second still arriving, the spare capacity is about 42,200 per second, so catch-up takes roughly 494 seconds, about 8 minutes.
This is why a durable log in front of processing matters: an outage becomes lag that clears, not data loss.
daily, sec = 500_000_000, 86_400
avg = daily / sec
peak = avg * 4
per_consumer = 2000
consumers = 24
backlog = avg * 3600
spare = consumers * per_consumer - avg
print(round(avg), round(peak))
print(round(peak / per_consumer, 1), "consumers at peak")
print(round(backlog), "events after a 1 hour outage")
print(round(backlog / spare), "seconds to catch up")Predict: roughly how many minutes does 24 consumers need to clear a 1-hour backlog?
5787 23148 11.6 consumers at peak 20833333 events after a 1 hour outage 494 seconds to catch up
Where the decisions come from
| Concern | Number that drives it | Decision |
|---|---|---|
| Burst absorption | Peak 23k events/s vs 5.8k average | Durable log between producers and processors; scale consumers by lag |
| Parallelism | About 24 consumers needed | 32 partitions |
| Hot keys | One key at 10% of peak is about 2,315/s, above one consumer's 2,000/s, so its lag grows by about 315/s | Detect with per-partition lag; salt the key only if per-key order can be relaxed |
| Small files | 100 GB per day of Parquet | See below |
| Bad records | Any malformed or oversized event | Dead-letter topic, so one bad record does not stall a partition |
| Schema change | Producers change independently of consumers | Schema registry with backward-compatible changes (add optional fields with defaults) |
| Cost | Retention x replication x peak sizing | Short Kafka retention, long lake retention; autoscale consumers |
The small-file trap in numbers
If 24 writers each commit a file every minute, that is 24 x 1,440 = 34,560 files per day, and 100 GB / 34,560 is under 3 MB per file. Query engines then spend more time opening files than reading them.
Committing every 10 minutes gives 3,456 files of about 29 MB, and a nightly compaction job that rewrites them toward 128 to 512 MB each (about 400 files per day) fixes the rest.
What to watch in production
- Consumer lag per partition, and the age of the oldest unprocessed event. These tell you how stale the data is.
- Dead-letter rate. A jump means a producer changed something.
- Duplicate rate at the sink, to prove your idempotent writes are doing their job.
- Freshness: time from event creation to visible in the warehouse, compared with the promised SLA.
Try a new scenario
A product ships 120 million events a day. Events average 3 KB, the busiest hour is 6x the average, Kafka keeps 7 days with 3 replicas, and each consumer handles 1,500 events per second. Compute average and peak events per second, peak bandwidth, Kafka disk before compression, and the minimum consumers at peak.
Work it out, then check
Average: 120,000,000 / 86,400 is about 1,389 per second. Peak: about 8,333 per second. Peak bandwidth: 8,333 x 3 KB is about 25 MB per second. Raw per day: 120 million x 3 KB = 360 GB, so Kafka disk is 360 GB x 7 x 3 = 7.56 TB before compression. Minimum consumers at peak: 8,333 / 1,500 is about 5.6, so 6, and with 2x headroom about 12. Choose 16 partitions to leave growth room.
The interviewer says traffic could be 10x on launch day. What do you change?
Re-run the arithmetic with the new peak, then decide which part must be pre-provisioned and which can autoscale. The log partition count is hard to change later, so size it for the larger peak. Consumers can scale out by lag. Check broker bandwidth, disk, and the sink's write limit, because the slowest component sets the real ceiling.
Close the lesson and explain it
Without scrolling up, explain how a daily volume turns into peak rate, partitions and storage. Then answer: why do we size for peak and not for the average, and what happens to lag when we do not?
Check yourself
Average-sized systems fall behind whenever traffic exceeds their capacity, and lag grows for as long as the surplus lasts. If the buffer is durable the data is still there afterwards, but freshness suffers and the backlog needs spare capacity to clear.
Finished this lesson?
Mark it as read when you could explain it to someone else.