Lesson 2 of 2 in this module

2/13 in track

Why a queue

python12 min

Overview

A click should not wait for the warehouse. A queue decouples the two, buffers when the reader is down, and lets you replay old messages.

On this page8 sections

The idea

A customer on ShopKart in Bengaluru taps Buy Now on a jacket.

Behind that single tap, four downstream systems need the order details: the warehouse to pack it, the notification service to send a confirmation SMS, the fraud engine, and the analytics database that powers the live revenue dashboard.

The straightforward approach is for the checkout server to make a direct network call to each downstream service in turn, waiting for each reply before confirming the order to the shopper.

That synchronous chain is brittle. If the analytics database is slow or undergoing maintenance, the customer's checkout screen spins. If an unhandled timeout occurs, the purchase can fail even though the customer's card was already charged.

A message queue solves this problem by placing a durable buffer between the producer and the consumers. The checkout server, acting as the producer, appends 1 message such as 'order ORD-1 placed' into the queue and immediately returns a success response to the customer.

Each downstream service acts as an independent consumer, reading messages from the queue at its own pace without blocking checkout or each other.

Producer, queue, consumer
Click event(producer)Message queue(buffer)WarehouseconsumerGold table

The producer drops a message and leaves. The consumer reads when ready. The queue holds everything in between.

Notice that only the producer step sits on the customer's critical path. Downstream consumers can slow down, crash, or restart for deployments, and the checkout page continues operating smoothly.

Think of a postbox at an apartment gate. The courier drops parcels into the box and leaves immediately. You collect your parcel when you return home in the evening. Neither party waits on the other.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
KafkaA system that stores a stream of events for many readers
TopicA named stream of events
ProducerWhatever writes events in
ConsumerWhatever reads events out
Consumer lagHow far behind the reader is
Poison pillA bad event that keeps crashing the reader

Why this exists

Placing a message queue between services provides three architectural benefits that direct API calls cannot match.

Decoupling: one slow system cannot freeze others

Suppose the customer notification service crashes at 21:00. With a message queue, the customer still sees 'Order confirmed', the warehouse still packs the jacket, and the payment settles.

The SMS receipts are simply delayed. When the notification service recovers, it resumes reading from where it stopped and sends the pending messages.

Buffering: absorbing traffic spikes

During a Diwali flash sale, order volume can surge from 50 orders per minute to 5,000 orders per minute. A database handling direct HTTP calls would be overwhelmed and drop connections.

A queue absorbs the traffic spike, within its storage and retention limits. The warehouse consumer keeps reading orders at a steady rate it can handle, which protects the database from the spike.

A queue buffers temporary spikes, but it does not make overload disappear. If arrivals exceed consumer capacity over a long period, unread messages accumulate. We inspect how to track this in the consumer lag section below.

Replay: re-reading historical events

Direct HTTP requests vanish from memory once delivered. By contrast, a persistent message log keeps messages on disk even after consumers read them.

Suppose a new tax calculation deployed on Tuesday contains a bug that is discovered on Thursday. You deploy the fix, rewind the consumer bookmark back to Tuesday morning, and reprocess every order to correct the warehouse table.

Multiple independent readers

When marketing requests order data for a new customer tool, the checkout service requires zero code changes. Engineers simply attach a new consumer to the existing queue.

The fraud engine, warehouse worker, analytics pipeline, and marketing loader each read from the same stream at their own independent speeds.

3 core jobs a queue performs that direct calls cannot.

AbilityWithout a queueWith a queue
DecoupleThe checkout page waits for every downstream writeCheckout writes 1 message and returns immediately
BufferTraffic spikes overload downstream services or drop eventsSurges wait safely in the queue until workers consume them
ReplayDelivered calls vanish; re-reading requires custom sender logicRewind the consumer bookmark and re-read historical messages

Picture this

Consider how the Wikimedia Foundation operates background updates across Wikipedia. When you save an edit on an article, MediaWiki must update every related page that links to it. That intensive work is not performed while your browser waits.

A diagram titled Kafka Job Queue Architecture. MediaWiki posts an event to an EventBus service in a Kafka cluster, and a change-propagation service in Kubernetes reads from Kafka and calls MediaWiki back to run the job
How the Wikimedia sites queue background jobs. MediaWiki posts a job (top left), the Kafka cluster holds it (top right), and a separate service reads it and tells MediaWiki to run it (bottom). The program that saves your edit never waits for the job itself.Source: Pchelolo, Wikimedia Commons · CC BY-SA 4.0

Follow the arrows in the diagram clockwise from the top-left box. MediaWiki publishes a job event to the Kafka cluster and returns to the editor immediately.

Later, an independent Kubernetes service reads the event from Kafka and calls MediaWiki back asynchronously to execute the update. The user who saved the edit never waits for the background processing.

A small example

In this track, a Python class called FakeKafkaTopic models the core behavior of an event queue. It wraps an in-memory list with two primary operations: producing and consuming.

Note that no Kafka broker runs in this tab. The exercises run standard Python lists. Real production systems like Apache Kafka or Amazon SQS run distributed clusters across many servers, but the foundational mechanics are identical.

Python
class FakeKafkaTopic:
    def __init__(self):
        self.messages = []

    def produce(self, message):
        """The sender adds a message to the end and moves on."""
        self.messages.append(message)

    def consume(self, offset, count):
        """The reader asks for count messages starting at offset."""
        return self.messages[offset:offset + count]

topic = FakeKafkaTopic()
topic.produce({"event": "click", "user": "u1", "page": "/products"})
topic.produce({"event": "click", "user": "u2", "page": "/cart"})
topic.produce({"event": "click", "user": "u1", "page": "/checkout"})

print(topic.consume(offset=0, count=2))
print(topic.consume(offset=2, count=5))
print(topic.consume(offset=0, count=3))
[{'event': 'click', 'user': 'u1', 'page': '/products'}, {'event': 'click', 'user': 'u2', 'page': '/cart'}]
[{'event': 'click', 'user': 'u1', 'page': '/checkout'}]
[{'event': 'click', 'user': 'u1', 'page': '/products'}, {'event': 'click', 'user': 'u2', 'page': '/cart'}, {'event': 'click', 'user': 'u1', 'page': '/checkout'}]

Observe how the three consume calls behave. 3 messages were buffered in memory before any consumer read them.

The first read fetched 2 messages starting at offset 0. The second read started at offset 2 and read to the end of the available list.

The third read started at offset 0 again and retrieved all 3 messages. Reading did not delete or destroy the messages. That persistence is what makes replay possible.

Here is the notification service outage represented as an offset trace. The service was offline while payments ORD-2 and ORD-3 arrived.

Messages at offset 1 and 2 arrived while the reader was offline.

offsetmessagenotification service status
0payment ORD-1 succeededprocessed before maintenance
1payment ORD-2 succeededbuffered in queue during outage
2payment ORD-3 succeededbuffered in queue during outage

The payment checkout service was never impacted. When the notification service restarted, it queried the queue starting from offset 1, sent the two delayed receipts, and caught up completely.

Monitoring consumer lag

To verify that consumers keep up with incoming traffic, monitor consumer lag: the difference between the newest message offset written by producers and the last offset processed by the consumer.

  1. Newest message: offset 1,000,000
  2. Consumer finished: offset 950,000
  1. Lag: 50,000 messages

Lag hovering near zero indicates consumers are keeping pace. Lag that increases steadily across hours indicates incoming volume exceeds consumer processing capacity, requiring faster processing or horizontal scaling.

Handling poison pills

Suppose a message contains malformed JSON or invalid types. A consumer reading it throws an exception, crashes, restarts, reads the same message again, and crashes in an infinite loop. This unprocessable record is known as a poison pill.

Production pipelines protect themselves using a dead-letter queue (DLQ). After a bounded number of retry attempts, the consumer moves the failing message to the dead-letter queue and advances its offset bookmark, allowing normal processing to continue.

Watch outQueues are buffers, not permanent warehouses

Queues move events between systems. In Apache Kafka, messages are retained for a configurable period (7 days by default) and then deleted. In Amazon SQS, messages are deleted once acknowledged. Consumers must persist processed results to durable stores like PostgreSQL or a data lake.

Common beginner questions

Why not use direct HTTP calls between all microservices?

Direct HTTP calls work well when the caller requires an immediate response to proceed. For handing off asynchronous pipeline tasks, direct calls tightly couple services: if the receiver is slow or down, the sender stalls, and failed requests cannot be replayed later.

Is a traditional message queue different from an event log like Kafka?

Yes. Traditional message queues (such as Amazon SQS or RabbitMQ) delete messages once a worker acknowledges them. Distributed event logs (such as Apache Kafka or AWS Kinesis) retain messages on disk for days or weeks, allowing multiple consumers to replay history from any offset bookmark.

Can a message queue run out of disk space?

Yes. If producer traffic continuously exceeds consumer throughput, unread messages accumulate. Distributed logs mitigate this by enforcing retention policies based on time or disk size, deleting old records once limits are reached.

What happens if a producer changes the message structure?

Producers and consumers deploy independently, so a renamed field can silently break downstream jobs. Engineering teams establish schema contracts using tools like Schema Registry, ensuring backwards compatibility before new message formats are published.

One purchase, told through a queue
  1. Checkout: A customer completes checkout on ShopKart. The server charges the card and saves the transaction.

    1. tap
    2. charge
    3. save order
  2. Producer: Checkout publishes an 'order ORD-1 placed' message to the queue and returns immediately to the shopper.

    1. message
    2. queue
    3. response
  3. Queue: The analytics consumer is experiencing high load, so the message waits safely in the queue buffer.

    1. downstream busy
    2. message buffered
  4. Consumer: The notification service restarts after maintenance, reads from offset 1, and dispatches the pending receipts.

    1. restart
    2. read offset 1
    3. receipts sent
  5. Replay: A downstream tax bug is fixed. The consumer rewinds its bookmark to the start of the week and re-reads historical records.

    1. deploy fix
    2. rewind offset
    3. reprocess

What comes next

One queue read by a single consumer works for moderate traffic. When messages arrive faster than one consumer process can handle, the queue is split into parallel lanes called partitions. That is the next lesson, on topics, partitions and keys.

Practice

Run the sample, then open the exercise. 3 problems are listed. For each one, decide which job of a queue solves it: decouple, buffer or replay. Put the 3 words in a list called result, in the same order, and print it.

Going further: a queue turns a burst into a wait

A queue sits between a producer and a consumer so that a burst does not have to be handled at the instant it arrives. The consumer works at its own steady speed, and the queue grows while the burst lasts and shrinks afterwards. Predict how deep the queue gets.

PythonA burst arrives faster than the consumer can work
arrivals = [10, 10, 100, 100, 100, 10, 10, 10, 10, 10]    # events per second
capacity = 40                                              # events the consumer handles per second

queue = 0
for second, arriving in enumerate(arrivals):
    queue = max(0, queue + arriving - capacity)
    print(f"second {second}: arrived {arriving:3}  queue depth {queue:3}")
second 0: arrived  10  queue depth   0
second 1: arrived  10  queue depth   0
second 2: arrived 100  queue depth  60
second 3: arrived 100  queue depth 120
second 4: arrived 100  queue depth 180
second 5: arrived  10  queue depth 150
second 6: arrived  10  queue depth 120
second 7: arrived  10  queue depth  90
second 8: arrived  10  queue depth  60
second 9: arrived  10  queue depth  30

The queue climbs to 180 during the burst and then shrinks by 30 a second (40 handled minus 10 arriving). It is still at 30 in the last second shown and reaches 0 1 second later, because the consumer's capacity (40) is above the normal arrival rate (10).

Nothing was lost, and nobody had to size the consumer for the 100-per-second peak. The cost is delay: events at the back of the queue waited several seconds.

The condition for this to work is that the average arrival rate stays below capacity. If arrivals stay above 40, the queue grows forever and the delay grows with it. That is when you add consumers or slow the producers.

Without a queueWith a queue
A burst overloads the consumer or is droppedThe burst waits in line
Producer and consumer must run at the same momentThey can run at different times, and the consumer can restart
One consumer only, usuallySeveral consumers can read the same events independently

Try a new scenario

Events arrive at a steady 60 per second for an hour, and your consumer handles 50 per second. What happens to the queue, and what are your options?

Work it out, then open the answer

The queue grows by 10 events a second. After an hour it holds 36000 events, and the delay keeps growing. A queue cannot fix a consumer that is slower than the average arrival rate. Add worker consumers, speed up the handler, or reduce what is sent. (A later lesson shows how partitions let multiple consumers share the work).

What is backpressure?

It is what happens when a consumer cannot keep up with producers. A queue absorbs short mismatches, but if the gap is permanent the queue grows without limit. Systems respond by slowing producers, adding consumers, or dropping data on purpose.

Close the lesson and explain it

Without scrolling up, explain how a queue absorbs a burst, and the one condition that makes it work.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?