System design interview

Disaster Recovery for Multi-Region Pipeline

HardLocked55 min read

DR for multi-region pipelines: cross-region Kafka/S3 replication, failover, RTO/RPO.

system-designdisaster-recoverymulti-regionreliability

Start here

The problem in plain words

Imagine your whole data platform lives in one city's data centre, and that city loses power for a day. Every pipeline stops and every dashboard goes blank. The question is not whether this could happen, but how quickly you can be running somewhere else, and how much recent data you would have lost.

Those two questions have names. How much recent data can we afford to lose is the RPO. How long can we be down is the RTO. A disaster recovery plan starts by agreeing both numbers with the business, because the numbers decide how much the design costs.

The usual answer is a second region that holds a copy of your data and can take over. Copying the bytes is the easy part. The hard parts are switching everything over safely, making sure the old region does not wake up and also write data, and practising so that it works on the bad day.

Words you will meet

Disaster recovery (DR)
The plan and setup for getting back to work after a whole region or site fails.
Region
A geographic cluster of cloud data centres, such as Mumbai or Ireland.
RPO
Recovery point objective: the most recent data you can afford to lose, measured in time, such as 15 minutes.
RTO
Recovery time objective: how long you can be down before service must be back, such as 2 hours.
Replication / mirroring
Continuously copying data from one region to another.
Active-passive
One region does the work, and a second stays ready and takes over only if the first fails.
Failover
Switching the work from the failed region to the backup one.
Failback
Moving the work back to the original region once it recovers.
Split-brain
Both regions think they are in charge and write at the same time, so the data diverges.
Fencing
Deliberately stopping the old region from writing, so only one region is in charge.
Offset
A consumer's bookmark for how far it has read in a Kafka topic.
DR drill
A planned practice run of the failover.
The primary region disappears
copiesarrivesRegion Aprimary, now downMirrorcopies Kafka and S3Region Bwarm backup
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.Normally, everything written in region A is copied across to region B, a little behind.
  2. 2.Region B holds almost everything, except the last few minutes still in flight when A failed.

The numbers, as things you can picture

  • RPO 15 minutes, RTO 2 hoursYou may lose up to 15 minutes of data and be down for up to 2 hours. These two numbers drive the whole design and its cost.

What the interviewer wants

Design disaster recovery for a multi-region data pipeline, with explicit RPO and RTO, a failover runbook, and a way to practise.

Interview room

DR for multi-region pipelines: replication, failover, RTO/RPO. Numbers before logos.

Basics

RPO/RTO definitions; active-passive vs active-active; replicate data+offsets+configs; drills required.

Prompt

Numeric targets, cross-region mirror, failover, fencing, cost, drills.

Good

Numbers; diagram; runbook; failback; game days.

Clarify/scope

Active mode; RPO minutes vs hours; residency.

Scoring lens: Disaster Recovery for Multi-Region Pipeline

Interviewers reward incident-first teaching, a clear control loop, and named owners. Tool logos without that loop score poorly.

Weak vs strong answers on Disaster Recovery for Multi-Region Pipeline

Weak: list products. Strong: tell the failure story, define the mechanism, draw the path, state rollout and monitors.

Scope control

Say what is out of scope for v1. Come back only if follow-ups demand it. This is a seniority signal for Disaster Recovery for Multi-Region Pipeline.

Clarifying then committing

Ask a few high-use questions about SLAs, owners, and scale, then sketch. Endless questions without a diagram look evasive.

Control-plane checklist

Images, secrets, IAM, DNS, job configs, schema registry connectivity in region B. Data bytes without control plane readiness fail drills.

Idempotent sinks and DR

Approximate offset restart is survivable if sinks upsert idempotently. Connect DR talk to everyday effectively-once design.

Failback is a second incident

Plan catch-up of A from B, drain, and cutover. Untested failback creates the sequel outage.

Interview framing detail 1

For lb-sd-35, spend the first minutes making the problem concrete: who gets hurt when this fails, what SLA is implied, and what is explicitly out of scope. Then teach the prerequisite concept before proposing boxes. A candidate who names vendors first usually loses the plot. Write two clarifying questions on the board and answer them with assumptions if the interviewer shrugs. Keep the first diagram small enough to redraw when a follow-up changes a constraint.