Lesson 1 of 3 in this module

3/9 in track

Ingest at 500M events/day

python18 min

Overview

Source volume, fan-in, Kafka versus files, and backpressure. Streaming already taught partitions and at-least-once. Here you size the front door.

On this page8 sections

The picture

Suppose a ShopKart flash sale makes orders arrive at 8 times the normal rate for 30 minutes. If the pipeline's front door, where raw events first enter, was sized for the average day, its buffer fills up and checkout has to wait for it.

The front door of a data pipeline must be sized for the busiest hour, not the average one.

This lesson sizes that door for 500 million events a day, using the workload brief established in [Requirements to numbers](/learn/system-design/design-req) and [Batch vs stream](/learn/system-design/design-batch-stream).

Fan-in to one buffer
checkoutpixelvendorbuffer

3 sources, one contract. The buffer is a queue or an object prefix if you stayed batch.

Fan-in means several sources, here checkout, a tracking pixel and a vendor feed, merging into one buffer without overwhelming whatever reads from it.

The streaming track taught topics, partitions, keys and at-least-once delivery. This lesson sizes the front door those lessons described. No broker runs in this tab.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
IngestTake data in from the source
ProducerWhatever sends events in, like an app
Partition keyThe value that decides which lane an event goes to
BackpressureWhen the receiver is slower than the sender
SinkWhere the data ends up
SchemaThe agreed list of fields and their types

Why this shape

The ingest layer decides how much can run in parallel, whether order is kept, and what happens when the sink is slow. Too few partitions stall at the peak. One hot key stalls one lane. Assuming the sink is infinitely fast invites an outage.

Bytes matter as much as counts. A 2 KB JSON event and a 200 byte binary event are very different loads at 23,000 events a second. Size for the peak hour, not the daily average.

Work it out for the flash sale. If the average day is 5,800 events a second and the sale multiplies that by 8, the door must absorb about 46,000 a second for half an hour.

A buffer sized for the average fills up in seconds and the checkout page times out.

Walk the boxes

Teaching arithmetic. Real peaks follow product launches. Size for the hour, not the average.

SliceEventsRough rate
Day500M~5.8k/s
Peak hour (4x)~83M~23k/s
Quiet hour (0.3x)~6M~1.7k/s
1 KB event, peak~23 MB/s inbound

Files or a queue? Follow the source. If checkout publishes to a topic, meet it there. If a vendor drops gzip files in a bucket, list the folder. Turning a file into a stream, or a stream into a file, adds work nobody asked for.

Two front doors
FilesQueueingestHourly dumpgs://bronze/dt=...Scheduler lists prefixBackfill = rerun datesLive producetopic eventsConsumer groupReplay = offsets

Object prefixes for file-based sources. Topics for live service producers.

The number of partitions limits parallelism, and the key pins one entity to 1 partition so its events stay in order. Choose a key with very many distinct values, such as user_id or session_id, never one with a few values such as country.

Lanes, not one pile
p0 user Ap1 user Bp2 user C

Same topic, 3 partitions. Key = user_id so one person's clicks stay ordered.

A matching example

Size the topic for 23,000 events a second at the peak, choose user_id as the key, and decide what happens when the sink falls behind.

High-cardinality user_id spreads load. Country would hot-spot India.

event_iduser_idcountrypartition key
e-001u-8821INu-8821
e-002u-4410USu-4410
e-003(none)INUUID fallback

Predict first: if 1 partition handles about 5,000 events a second, how many do you need at 23,000?

PythonPartition key choice and sizing arithmetic
import math

def choose_partition_key(event):
    """Pick a high-cardinality key that distributes evenly."""
    if "user_id" in event:
        return event["user_id"]
    # Bad key: country has ~200 values and creates hot partitions
    return event.get("session_id", "fallback-uuid")

def estimate_partitions(peak_rate, target_per_partition=5000):
    """Size partition count for peak throughput, rounding up."""
    return max(math.ceil(peak_rate / target_per_partition), 3)

print(estimate_partitions(23_000))
print(estimate_partitions(1_000))
5
3

23,000 divided by 5,000 is 4.6, which rounds up to 5. Rounding down to 4 would leave you short at the peak, so always round up, and add headroom. The floor of 3 is a sensible minimum even for a quiet topic.

Watch outHot keys stall parallelism

If a large share of traffic shares one partner id, that partition becomes the bottleneck. Salt the key or isolate that partner on its own topic.

What if one key is much bigger than the rest?

Say one big merchant generates 40% of all orders and you chose merchant_id as the key. Partition 3 drowns while the others sit idle.

Two fixes: add the order id to the key so the merchant's orders spread out, or add a small salt of 0 to 7 so one hot key becomes eight.

PythonOne hot merchant spread over eight lanes
def salted_key(merchant_id, order_id, buckets=8):
    return f"{merchant_id}_{sum(ord(c) for c in order_id) % buckets}"

orders = [f"ORD-{i}" for i in range(1, 9)]
print(sorted({salted_key("M-1", o) for o in orders}))
['M-1_0', 'M-1_1', 'M-1_2', 'M-1_3', 'M-1_4', 'M-1_5', 'M-1_6', 'M-1_7']

The merchant now lands on eight different keys. The cost is that something downstream must add the eight pieces back together, usually with a 2-step aggregation.

What happens when the sink is slower than the producers?

That is backpressure. You have three choices. Block the producers, which is honest if they retry and the sink ignores repeats.

Drop events, which needs an explicit budget and an alert. Or spill to a dead-letter queue, which is a leak unless a job drains it.

  • Block: producer retries; at-least-once will fire; idempotent sink required.
  • Drop: only with an explicit budget and a page threshold.
  • Spill / DLQ: quarantine, count, drain on a schedule.
PythonBackpressure strategy skeleton
# Backpressure strategy skeleton
class IngestBuffer:
    def __init__(self, max_queue_size: int = 100_000):
        self.max_queue_size = max_queue_size
        self.dlq_count = 0

    def accept(self, event: dict) -> str:
        if self.queue_size() < self.max_queue_size:
            self.enqueue(event)
            return "accepted"
        elif self.can_drop(event):
            self.increment_drop_metric()
            return "dropped"  # only low-priority events
        else:
            self.send_to_dlq(event)
            self.dlq_count += 1
            if self.dlq_count > 10_000:
                self.page_oncall("DLQ growing fast")
            return "quarantined"
Watch outDo not assume infinite sink capacity

The database, the warehouse, and the queue all have throughput limits. Size the sink write path for peak rate, not average. Test what happens when the sink is temporarily down.

RememberAt-least-once is the default at 500M

Assume duplicates. Design the sink as if every event may arrive twice. Orchestration idempotent-load is how you survive that assumption.

RememberCore Python retries

You practiced exponential backoff in the Core Python track. The same pattern applies to ingest producers that get a 503 from the buffer. Retry with backoff, then page if retries exhaust.

Which format should events use on the wire?

At 500 million events a day the format changes the network bill. JSON is easy to read but large. Binary formats are smaller and enforce a schema.

Serialization format trade-offs at 500M daily event scale.

FormatAverage Payload SizeSchema EnforcementCPU Deserialization Cost
JSON1,200 bytesImplicit / Runtime validationHigh (string parsing, unicode decoding)
Apache Avro220 bytesStrict (Confluent Schema Registry)Low (binary compact encoding)
Protocol Buffers180 bytesStrict (compiled IDL definitions)Very low (optimized native binary bindings)

At 1,200 bytes, 500 million events are about 600 GB a day. At 180 bytes they are about 90 GB. A schema registry also stops a producer from shipping a breaking change to a live topic.

Protecting the door at the edge

TechniqueWhat it doesPlain example
Rate limit (token bucket)Gives each client a quota and answers HTTP 429 beyond itA buggy partner app cannot flood you
Circuit breakerPast 90% queue capacity, sheds low-value telemetry and keeps purchasesPage views are dropped before payments are
Backoff with jitterClients wait base * 2**attempt plus a random amount between retriesThousands of retrying phones do not arrive in a synchronized wave
Compression (LZ4, zstd)Shrinks the bytes on the wire at a small CPU costLZ4 can cut JSON traffic by about 65%
Regional entry pointsAccept traffic near the user, then ride the cloud's own networkA phone in Singapore does not cross the public internet to one region

The backoff schedule is easy to see. With a base of 0.1 seconds and a cap of 5, successive retries wait 0.1, 0.2, 0.4, 0.8, 1.6, 3.2 and then 5 seconds, with a little randomness added each time.

Common beginner questions

What if I only have 1 million events per day, not 500 million?

The same principles apply at smaller scale, but the urgency is lower. At 1 million per day you can get away with simpler tooling (a single file dump, a small queue). The partition math just yields fewer partitions. Learn the pattern at 500M so you can scale down confidently.

Do I need Kafka specifically?

No. Kafka is one implementation of a distributed message queue. Google Pub/Sub, Amazon Kinesis, Azure Event Hubs, and Redpanda all fill the same architectural slot. The design brief cares about the pattern (queue with partitions and consumer groups), not the vendor.

What happens if the system gets 10x more data overnight?

If your partition key is high-cardinality, you add more partitions and more consumer instances. If you hardcoded a low-cardinality key (like country), you are stuck. Partition key choice is the decision that is hardest to change later.

Sizing the door for the flash sale
  1. Problem: A flash sale pushes orders to 8 times normal. A buffer sized for the average fills in seconds, and checkout times out.

    1. 8x spike
    2. buffer full
    3. checkout hangs
  2. Rate: Start from the brief: 500M events a day is 5,800 a second on average and about 23,000 at the 4x peak. Size the door for the peak.

    1. 5.8k/s avg
    2. 23k/s peak
  3. Door: Pick the door by the source: a topic for live services, an object folder for vendor files. Choose user_id as the key so one person's events stay in order.

    1. topic or files
    2. key = user_id
  4. Sizing: estimate_partitions(23_000) is 4.6, rounded up to 5. Rounding down would leave the door short at the peak.

    1. 23,000 / 5,000
    2. round up
    3. 5
  5. Hot key: One merchant with 40% of orders is a hot key. Salt it across eight lanes and add the pieces back downstream.

    1. M-1
    2. M-1_0..7
  6. Backpressure: When the sink is slow, block with retries, drop with a budget or spill to a drained dead-letter queue. Design the sink for duplicates.

    1. block
    2. drop
    3. DLQ

What comes next

The next lesson covers storage layout: how to land those 500 million events into bronze, silver, and gold layers so that downstream jobs can prune to 1 day without scanning the entire lake.

Practice

Run estimate_partitions(23_000) and confirm the answer is 5. Then write one sentence: why country is a bad partition key at this volume.

Going deeper: sizing 500 million events a day from the requirements

A design for 500 million events a day should read like arithmetic that ends in decisions, not a list of technologies. This section does that arithmetic step by step, states every assumption, and shows which decision each number drives. Change an assumption and the numbers change; that is the point.

State the assumptions first

AssumptionValue used hereWhy this is a guess you should confirm
Events per day500,000,000Given
Average event size1 KB (JSON, uncompressed)Measure real payloads; one large field changes everything
Peak to average ratio4xLook at a real traffic curve; a sale or a push notification can be far higher
Kafka retention and replication7 days, 3 copiesRetention sets how far back you can replay; replicas buy durability
Consumer speed2,000 events/s per consumer instanceBenchmark your actual handler; this depends on the work done per event
Lake compressionParquet about 5x smaller than raw JSONDepends on the data; measure a sample

Step 1: rate

A day has 86,400 seconds. Size for the peak, not the average.

QuantityCalculationResult
Average events per second500,000,000 / 86,400about 5,787 per second
Peak events per second5,787 x 4about 23,148 per second
Average bandwidth5,787 x 1 KBabout 5.8 MB per second
Peak bandwidth23,148 x 1 KBabout 23.1 MB per second

Step 2: storage

Retention and replication multiply the disk bill, so they are the first cost lever.

PlaceCalculationResult
Raw data per day500,000,000 x 1 KB500 GB
Kafka on disk (7 days, 3 copies, before compression)500 GB x 7 x 310.5 TB
Kafka with 4x compression10.5 TB / 4about 2.6 TB
Lake (Parquet, 5x)500 GB / 5100 GB per day, about 36.5 TB per year

Step 3: parallelism

At 2,000 events per second per consumer, peak needs 23,148 / 2,000, about 11.6 consumers. Add 2x headroom for retries, deploys and a slow broker, and you get about 24.

Kafka cannot give more consumers than partitions work, so choose at least 24 partitions, and round up (for example 32) so you can grow without repartitioning.

Step 4: recovery

Suppose the consumers are down for 1 hour. At the average rate, 5,787 x 3,600 is about 20.8 million events waiting in the log. With 24 consumers handling 48,000 per second and 5,787 per second still arriving, the spare capacity is about 42,200 per second, so catch-up takes roughly 494 seconds, about 8 minutes.

This is why a durable log in front of processing matters: an outage becomes lag that clears, not data loss.

PythonThe sizing in code
daily, sec = 500_000_000, 86_400
avg = daily / sec
peak = avg * 4
per_consumer = 2000
consumers = 24
backlog = avg * 3600
spare = consumers * per_consumer - avg
print(round(avg), round(peak))
print(round(peak / per_consumer, 1), "consumers at peak")
print(round(backlog), "events after a 1 hour outage")
print(round(backlog / spare), "seconds to catch up")

Predict: roughly how many minutes does 24 consumers need to clear a 1-hour backlog?

5787 23148
11.6 consumers at peak
20833333 events after a 1 hour outage
494 seconds to catch up

Where the decisions come from

ConcernNumber that drives itDecision
Burst absorptionPeak 23k events/s vs 5.8k averageDurable log between producers and processors; scale consumers by lag
ParallelismAbout 24 consumers needed32 partitions
Hot keysOne key at 10% of peak is about 2,315/s, above one consumer's 2,000/s, so its lag grows by about 315/sDetect with per-partition lag; salt the key only if per-key order can be relaxed
Small files100 GB per day of ParquetSee below
Bad recordsAny malformed or oversized eventDead-letter topic, so one bad record does not stall a partition
Schema changeProducers change independently of consumersSchema registry with backward-compatible changes (add optional fields with defaults)
CostRetention x replication x peak sizingShort Kafka retention, long lake retention; autoscale consumers

The small-file trap in numbers

If 24 writers each commit a file every minute, that is 24 x 1,440 = 34,560 files per day, and 100 GB / 34,560 is under 3 MB per file. Query engines then spend more time opening files than reading them.

Committing every 10 minutes gives 3,456 files of about 29 MB, and a nightly compaction job that rewrites them toward 128 to 512 MB each (about 400 files per day) fixes the rest.

What to watch in production

  • Consumer lag per partition, and the age of the oldest unprocessed event. These tell you how stale the data is.
  • Dead-letter rate. A jump means a producer changed something.
  • Duplicate rate at the sink, to prove your idempotent writes are doing their job.
  • Freshness: time from event creation to visible in the warehouse, compared with the promised SLA.

Try a new scenario

A product ships 120 million events a day. Events average 3 KB, the busiest hour is 6x the average, Kafka keeps 7 days with 3 replicas, and each consumer handles 1,500 events per second. Compute average and peak events per second, peak bandwidth, Kafka disk before compression, and the minimum consumers at peak.

Work it out, then check

Average: 120,000,000 / 86,400 is about 1,389 per second. Peak: about 8,333 per second. Peak bandwidth: 8,333 x 3 KB is about 25 MB per second. Raw per day: 120 million x 3 KB = 360 GB, so Kafka disk is 360 GB x 7 x 3 = 7.56 TB before compression. Minimum consumers at peak: 8,333 / 1,500 is about 5.6, so 6, and with 2x headroom about 12. Choose 16 partitions to leave growth room.

The interviewer says traffic could be 10x on launch day. What do you change?

Re-run the arithmetic with the new peak, then decide which part must be pre-provisioned and which can autoscale. The log partition count is hard to change later, so size it for the larger peak. Consumers can scale out by lag. Check broker bandwidth, disk, and the sink's write limit, because the slowest component sets the real ceiling.

Close the lesson and explain it

Without scrolling up, explain how a daily volume turns into peak rate, partitions and storage. Then answer: why do we size for peak and not for the average, and what happens to lag when we do not?

Check yourself

Average-sized systems fall behind whenever traffic exceeds their capacity, and lag grows for as long as the surplus lasts. If the buffer is durable the data is still there afterwards, but freshness suffers and the backlog needs spare capacity to clear.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?