System design interview

Event-Driven Order Processing System

HardLocked55 min read

Food delivery order flow with Kafka state transitions, transactional outbox, and timeout cancellations.

system-designevent-drivenkafkaoutboxorders

Start here

The problem in plain words

You order food. The platform saves your order, tells the restaurant, finds a driver and notifies you. If the saving works but the message to the restaurant is lost, you wait forever while the restaurant tablet stays silent. Nobody did anything wrong, but the order is stuck.

This happens because the system writes in two places, a database and a message queue, and a crash between those two writes leaves them disagreeing. The fix is a clever habit: write the order and a note saying "tell everyone" in the same database transaction, and let a separate helper deliver the note.

Add two more things and you have a reliable flow: consumers that can safely see the same message twice, and a timer that cancels an order the restaurant never accepts.

Words you will meet

Event-driven
Services communicate by publishing and reacting to events rather than calling each other directly.
State machine
A set of allowed states, such as Placed, Accepted, Cancelled, and the allowed moves between them.
Kafka topic
A named log of events that many services can read independently.
Dual write
Writing to a database and a queue separately. A crash between the two leaves them out of step.
Transactional outbox
A table written in the same transaction as the order, holding events waiting to be published.
Outbox publisher
A small process that reads the outbox and sends the events to Kafka.
Idempotent consumer
A consumer that gives the same result even if it receives the same event twice.
At-least-once delivery
A message is delivered one or more times, never zero, so duplicates are possible.
Timeout / delay queue
A way to say "do this later if nothing happened by then".
Dispatch
The service that assigns a delivery driver.
The order is saved, but the restaurant never hears
insertcrashsilenceOrder servicesaves the orderDatabaseorder savedKafkaevent never sentRestauranttablet stays quiet
Press Play to watch it move

Each arrow is one thing happening. Play it step by step, or read the list below.

  1. 1.The order is inserted into the database successfully.
  2. 2.The process dies before it publishes the OrderPlaced event, so Kafka never receives it.
  3. 3.The restaurant never learns about the order. The customer waits, and support has to fix it by hand.

The numbers, as things you can picture

  • About 50 orders a second at Friday dinnerBursts at meal times, so the flow must be reliable and cheap per order, not just correct at low load.

What the interviewer wants

Design a reliable order flow with a state machine, a transactional outbox, idempotent consumers and timeout cancellation.

The interview room

Design an event-driven order processing system for food delivery. Kafka carries state transitions like OrderPlaced, OrderAccepted, DriverAssigned. You must use the Outbox pattern so DB and Kafka stay consistent, and timeout events so unaccepted orders cancel cleanly.

Interviewers love this prompt because it mixes a state machine, messaging reliability, and real failure modes (restaurant never accepts, driver cancels, dual-write bugs).

What "event-driven" means here

Services react to facts ("order placed") instead of calling a giant synchronous orchestrator for every hop. The order service writes durable state. Other services (restaurant, dispatch, notifications) consume events and act.

The dual-write problem

Naive flow:

1. INSERT order into Postgres. 2. produce OrderPlaced to Kafka.

If you crash between 1 and 2, the DB has an order and nobody else knows. If you reverse the order and crash after Kafka, you have a phantom event with no DB row. Two non-atomic writes are the bug. This is the dual-write problem.

Outbox pattern (the fix)

In one database transaction:

  • Write the order row (or state transition).
  • Write an outbox row with the event payload.

A separate publisher reads outbox rows and produces to Kafka, then marks them published. DB commit is the atomic boundary. Kafka delivery becomes reliable asynchronous propagation.

State machines

Orders are not free-form strings. Legal transitions matter:

Placed --> Accepted --> DriverAssigned --> PickedUp --> Delivered
   \          \             /
    +-----> Cancelled <----+

Illegal jumps (Delivered → Placed) must be rejected. Timeouts are first-class events (OrderAcceptanceTimedOut) that drive cancels.

Idempotent consumers

At-least-once delivery means consumers may see duplicates. Processing OrderPlaced twice must not charge twice or assign two drivers. Key on event_id or (order_id, transition).

The problem you are designing

Food delivery order flow:

  • Persist order state transitions reliably.
  • Publish events without critical loss from dual-write races.
  • Timeout unaccepted orders.
  • Downstream consumers update logistics and notifications.
  • Regional scale with meal-time bursts.

What good looks like

You name dual-write as the core reliability bug. You draw outbox. You sketch a state machine. You add a timeout path. You mention idempotent consumers. You discuss conflict cases (driver assigned, then restaurant cancels).

Clarifying questions

  • Who is SoR for order state: the order DB, or a pure event-sourced log?
  • Timeout durations for restaurant accept and driver assign?
  • Sync response to the customer app vs eventually consistent confirmation?
  • Multi-city topics vs one big cluster?
  • Payment authorization timing relative to OrderPlaced?

Constraints

  • Meal peaks create bursty produce/consume.
  • Losing OrderPlaced is a Sev-1 for restaurants and customers.
  • Duplicate notifications are annoying; duplicate charges are catastrophic.

Out of scope

  • Full GIS routing algorithm for drivers.
  • Detailed payment network design (mention authorize/capture at high level).
  • Mobile app UX beyond API contracts.

Why food delivery is a great prompt

Orders cross organizational boundaries. The restaurant is not your database. The driver is not your database. Notifications are best-effort. Payments are strict. That mix forces you to talk about reliability classes, not one generic "event bus."

Dual-write in slow motion

Walk this with the interviewer:

1. API handler begins. 2. INSERT into orders succeeds. 3. Process is OOM-killed before producer.send. 4. Customer sees "order placed" from the API response that read the DB. 5. Restaurant never receives OrderPlaced. 6. Support invents a manual cancel 30 minutes later.

That story is worth more than naming five Kafka features.

Outbox mental model

Treat the outbox table as "events the business has decided are true but the world has not heard yet." The publisher is a reliable postman, not the decision maker. Decision making stays in the transactional order service.

Timeout as a first-class citizen

Without timeouts, Placed becomes a junk drawer. Restaurants forget. Customers wait. Inventory holds linger. A timeout event is not a niche feature. It is how you keep the state machine honest when humans do not respond.

What good looks like beyond the diagram

Mention idempotent consumers, per-order partitioning keys, outbox lag alerts, and one conflict policy (cancel after assign). If you only draw boxes, they will keep asking "what if" until those appear.

Reliability classes (teach explicitly)

Not every side effect needs the same guarantee:

  • Order state + payments: hard correctness, idempotent, audited
  • Restaurant ticket: durable, delayed is bad but recoverable
  • Push notification: best effort with debounce
  • Analytics copy of events: at-least-once into the lake is fine

Naming reliability classes prevents over-engineering notifications and under-engineering payments.

Burst and backpressure awareness

At dinner peak, dispatch consumers may lag. Kafka retains. You should say what the lag SLO is and when you page. Backpressure belongs in this design even if the prompt emphasizes outbox.