System design interview

Real-Time Fraud Detection (<100ms Latency)

HardFree75 min read

Design fraud detection for ~5k TPS payments with <100ms decision latency, including features, scoring, and feedback loops.

system-designstreamingfraudlow-latencyml

Start here

The problem in plain words

You tap your card to pay for a coffee. The card machine now waits for the bank to say yes or no. Somewhere in that wait, a fraud service has to decide whether this payment looks suspicious, and it only has about the time of a blink. That time is 100 milliseconds.

Someone says: "Just query the warehouse for the customer's last 90 days, then run a big model on a GPU cluster." That takes seconds. By then the customer has walked away, or the payment has timed out.

So the design is really three designs joined together. One answers instantly using only tiny, pre-prepared lookups. One records every decision calmly afterwards. And one learns slowly from what turned out to be fraud, sometimes weeks later. Mixing them into a single box is the classic mistake.

Words you will meet

TPS
Transactions per second. 5,000 TPS means five thousand payments each second.
Latency budget
The total time allowed for a response, here 100 ms, which every step must share.
p99
The time within which 99 out of 100 requests finish. Payments care about the slow tail, not only the average.
Network hop
One call from one service to another. Each hop adds delay, and the delays add up.
Online features
Ready-made values for one customer or card, such as number of payments in the last hour, that are fetched by key in milliseconds.
Rules engine
Simple if-then checks, such as blocking a card from a banned country, that run alongside or instead of a model.
Fail-open / fail-closed
What to do if the fraud service is broken: approve the payment anyway (open), or decline it (closed).
Idempotency
Handling a repeated request for the same payment the same way, so a retry gets the same decision.
Feedback loop
Using what really happened, such as confirmed fraud, to train a better model.
Delayed label
The true answer arrives late, for example a chargeback weeks after the purchase.
A decision that takes seconds is no decision
approve?historytoo lateCard machinewaits for yes or noFraud servicehas about 100 msWarehouse query90 days of history
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.A payment arrives and the card machine is waiting for the answer.
  2. 2.The naive design queries a warehouse for 90 days of history, then runs a big model. That alone takes seconds, not milliseconds.
  3. 3.By the time the answer comes back the payment has timed out. A decision path must only use lookups that are already prepared.

The numbers, as things you can picture

  • 5,000 payments a secondAbout 432 million a day if it stayed at that rate. Size the system for the busiest moment, not the average.
  • 100 ms at p99Nearly every decision, not just most, must return within a blink.
  • 1 to 3 ms for a nearby key-value lookupA cache in the same data centre zone is fast. A call to another region would use most of the budget.

What the interviewer wants

Design the instant approve or decline path, the way every decision is recorded, and the slow learning loop that improves the model, while keeping payments working even if parts of fraud detection break.

The interview setup

You are in a mid-level data engineering interview. The prompt sounds like machine learning platform work. The interviewer is actually testing something sharper: can you protect a synchronous payment path with a hard latency budget, while still building a learning loop that improves over time?

A customer taps a card at a coffee shop. Your payment API needs an approve, decline, or step-up answer in under 100 milliseconds at about 5,000 transactions per second. That is not "run Spark overnight and email Risk." That is a real-time decision service with an asynchronous training world wrapped around it.

What interviewers reward here:

  • You feel why 100ms changes the design
  • You separate the online decision path from the offline training path
  • You budget milliseconds on the whiteboard
  • You name fallbacks when features or models are unhealthy
  • You talk about delayed labels (chargebacks), not magically instant truth

What online fraud detection actually is

Fraud detection is a decision about a payment before money fully moves, or before capture completes. Inputs are the payment fields plus features: signals about the card, device, merchant, velocity, geography, and history.

Typical outputs:

  • Approve - let the payment through
  • Decline - hard stop
  • Review / step-up - OTP, 3-D Secure, or a human queue

You are not building a BI dashboard. You are building a gate on the money path. Product, finance, and risk all care, and they disagree about false positives.

Why latency is a product requirement, not a nice-to-have

Card networks and checkout UX have tight timeouts. If fraud takes 800ms, either payments fail or product removes your check. Healthy fraud platforms therefore split into layers:

1. Online path - tiny latency budget, nearby dependencies only 2. Nearline path - enrichment and heavier features that update profiles asynchronously 3. Offline path - training, evaluation, shadow scoring, rule analytics

If you remember one sentence for the rest of your career: never put a batch warehouse query on the authorization critical path.

Online vs offline feature stores

A feature is a computed signal used by rules or models: transaction count in the last hour, average ticket over 30 days, whether the device was seen before, merchant fraud rate over 7 days.

An online feature store serves precomputed or fast-computed features with low latency. Think Redis, Cassandra, DynamoDB, or a purpose-built feature service.

An offline feature store / training set stores historical feature values joined to labels for model training.

The interview killer is training-serving skew: you train on features that do not exist online, or that are computed differently online. The notebook looks brilliant. Production is blind.

Rules vs models

Mature stacks combine:

  • Rules - explicit, auditable, fast ("decline if country flip + high amount + new device")
  • ML models - score risk from many features
  • Policy - model score + rules + business thresholds become the final action

Rules are guardrails and kill switches. Models catch patterns rules miss. Strong candidates put rules as overrides, not as an afterthought.

Feedback loops and delayed labels

Truth for fraud is often late. A chargeback may arrive days or weeks after purchase. Your training pipeline must join decisions to outcomes by payment id with lag. Do not pretend labels arrive at decision time.

Human review outcomes and merchant reports are also labels, with different delay and noise. Good designs log the feature vector and model version at decision time so later labels can attach cleanly.

The expanded problem

Design fraud detection for a payment processor at roughly 5,000 TPS. Return a decision in under 100ms p99. Include feature fetch or computation, model scoring and rules, decision logging for audit and training, a feedback loop for retraining, safe rollouts, and kill switches.

Assume you own the decision service. Kafka (or similar) is fine for async logs. Object storage and a warehouse are fine for training. They are not fine on the sync path.

Constraints and scale prompts (say numbers out loud)

  • 5,000 TPS sustained is about 432 million transactions per day; size for peak, not average
  • Latency budget is scarce; feature store p99 and network hops dominate
  • Model location matters: embedded in-process vs remote model server in the same AZ
  • Idempotency: clients retry; the same payment id must not silently flip decisions

Example latency budget you can write on the board:

  • Network into decision service: 5–10ms
  • Feature fetch (batched): 20–40ms
  • Model inference: 5–20ms
  • Rules and policy: 5–10ms
  • Margin for GC and jitter: the rest

If feature fetch alone is 80ms, the design is already in trouble.

What good looks like in the room

A strong answer: 1. Clarifies hard decline vs step-up vs async review 2. Draws sync path and async training path as two systems that meet at a log 3. Breaks the 100ms budget explicitly 4. Names online feature store plus a degradation policy 5. Explains delayed labels and shadow or canary model deploy 6. Mentions audit trails for declines and a kill switch for bad models

Clarifying questions to ask

  • Is 100ms p99 end-to-end from the payment API, or fraud service internal only?
  • Which outcomes are allowed: hard decline, step-up, human review?
  • What false positive rate is tolerable? Declining good customers kills conversion.
  • Do we already have an online feature store, or design from scratch?
  • Single region or multi-region active-active payments?
  • Primary label source: chargeback, manual review, or both?
  • Regulatory audit needs for decline reasons?

Out of scope (say this out loud)

  • Inventing the "best" neural net architecture from scratch
  • A full PCI deep dive (mention awareness, do not boil the ocean)
  • Building the entire card network
  • Real-time graph neural nets unless the interviewer invites the stretch

Your job is the decision path and the data feedback loop, not a research paper.