Think of a restaurant kitchen. Orders arrive at the door faster than the cooks can finish them. The tickets pile up. If nothing changes, the pile grows until tickets are lost, or the kitchen collapses.
A streaming pipeline has the same moment. Producers send events faster than the consumers can process them, or the database at the end cannot keep up. Lag, the number of events waiting, climbs. This pressure that builds up is called backpressure.
A good design does not hope for the best. It has a plan with layers: find where the slow point is, add capacity there, let a buffer absorb the spike within limits, drop the least important work in a controlled way, and tell producers to slow down.
Words you will meet
Backpressure
The pressure that builds when a downstream part cannot keep up, pushing work back to the parts before it.
Consumer lag
How many events (or how much time of data) are waiting to be processed.
Throughput
How much work a part completes per second.
Buffer
A holding area for waiting work. A message log such as Kafka is a durable buffer with a limited retention time.
Retention
How long the log keeps events. If the consumer takes longer than that to catch up, data is lost.
Sink
The final destination, such as a database or warehouse, which is often the real bottleneck.
Autoscaling
Automatically adding workers when load rises, usually with a maximum.
Load shedding
Deliberately dropping or delaying low-priority work to protect the important work.
429 / pause
A signal telling producers to slow down or stop for a while.
SLO
A target for service quality, such as "lag under 60 seconds".
Events arrive faster than they can be handled
Press Play to watch it move
Each arrow is one thing happening. Play it step by step, or read the list below.
1.A burst of events arrives, much faster than normal.
2.Consumers read as fast as they can, but the backlog, the lag, keeps growing.
3.The real limit is the sink: its CPU is saturated, so adding more consumers would only make it worse.
The numbers, as things you can picture
Spike size against retentionIf the spike adds more events than the log can hold, or catch-up takes longer than retention, events are lost. Do this sum before choosing a plan.
What the interviewer wants
Explain backpressure and give a layered response: find the bottleneck, scale it, buffer safely, shed load by priority, and signal producers, with measurements that show it is working.
Backpressure = intentional slowdown/drop. Kafka lag is bounded buffer. Infinite memory queues are delayed OOM. Layer scale/buffer/signal/shed/priority.
Prompt
Define; Kafka semantics; scale vs shed; protect critical paths.
Good
Sink bottleneck story; four strategies; lag SLO; droppable classes.
Clarify/scope
Droppable?; max lag; critical event types.
Scoring lens: Backpressure in Streaming Pipeline
Interviewers reward incident-first teaching, a clear control loop, and named owners. Tool logos without that loop score poorly.
Weak vs strong answers on Backpressure in Streaming Pipeline
Weak: list products. Strong: tell the failure story, define the mechanism, draw the path, state rollout and monitors.
Scope control
Say what is out of scope for v1. Come back only if follow-ups demand it. This is a seniority signal for Backpressure in Streaming Pipeline.
Clarifying then committing
Ask a few high-use questions about SLAs, owners, and scale, then sketch. Endless questions without a diagram look evasive.
Priority topic design
events.payments and events.debug are different products. Different lag SLOs and shed policies. Putting everything on one topic forces bad global decisions.
Catch-up capacity
Buffers only help if sink rate can exceed steady state after a spike. Say the catch-up math. Otherwise retention merely delays the outage.
Tenant fairness under shed
Shed or throttle the noisy tenant first on shared platforms. Global random drops punish the innocent.
Interview framing detail 1
For lb-sd-32, spend the first minutes making the problem concrete: who gets hurt when this fails, what SLA is implied, and what is explicitly out of scope. Then teach the prerequisite concept before proposing boxes. A candidate who names vendors first usually loses the plot. Write two clarifying questions on the board and answer them with assumptions if the interviewer shrugs. Keep the first diagram small enough to redraw when a follow-up changes a constraint.