Lesson 1 of 3 in this module

3/13 in track

Topics, partitions, keys

python14 min

Overview

A topic is a named log split into partitions. Keys normally choose a partition so related events stay together, but the exact partitioning rule belongs to the producer client.

On this page8 sections

The picture

Picture one billing counter at a supermarket on a festival evening. Hundreds of people stand in a single line, and one cashier serves them. The shop opens eight more counters and the line finally moves.

A queue has the same problem. ShopKart sends 100,000 click events a second into one queue, and a single reader cannot keep up. So the queue is cut into lanes, called partitions, and a separate reader takes each lane.

There is a catch. If the 3 events of one order (created, paid, shipped) land in three different lanes, three readers may handle them at the same moment, and shipped can be processed before paid.

So we need a rule: all events of one order go to the same lane.

Same key, same partition
P0Message key: ORD-1P1Add up character codes: 323P2323 % 3 partitions = 2P3Append to partition 2

ORD-1 always lands in partition 2, so every event about ORD-1 sits in one list, in the order it was written.

The four boxes are how 1 message finds its lane. It carries a key (ORD-1). The key is turned into a number, the number is divided by 3, and the remainder picks the lane. That is all the routing there is.

Diagram with 3 producers on top sending into a Kafka cluster that contains 3 topics, each topic split into 3 partitions, with 3 consumers reading from the cluster below
Producers write into topics, each topic is split into partitions, and consumers read from them. Everything in the middle box lives on the Kafka cluster.Source: Ch.ko123, Wikimedia Commons · CC BY 4.0

This is the whole system in one frame. The boxes on top are programs that write. The boxes at the bottom are programs that read.

In the middle, the cluster holds topics, and every topic is a stack of partitions. The words topic and partition are what this lesson is about.

Words you will see in this lesson

Six words. The rest of the lesson uses only these.

WordWhat it means
PartitionOne lane of a topic
KeyThe value that decides which lane an event goes to
ConsumerWhatever reads events out
LaneAnother word for one partition
OrderEvents with the same key stay in the order they arrived
TopicA named stream of events

Why this shape

Two separate problems push you to this design.

The first is speed. If one consumer can handle 10,000 messages a second and 100,000 arrive, you need 10 consumers.

They cannot all read from the same list without stepping on each other. So the list is split into partitions, and each partition is read by one consumer at a time.

The second is order. Suppose ORD-1 produces 3 events: created, paid, shipped. If those went to three different partitions, three different consumers might process them at the same moment. "Shipped" could be handled before "paid", and the order's status would end up wrong.

So every event for ORD-1 must go to the same partition. The key is what makes that happen.

You pay for it in one place: order is only guaranteed inside a partition. 2 events for different orders may be read in either order, and for different orders, that rarely matters.

A real Kafka setup hashes the key with an algorithm called murmur2 to pick a partition.

In this track we use a simpler stand-in, the sum of the key's character codes, because you can work it out by hand. The behaviour is the same: a given key always gives the same partition.

Walk the boxes

Here is the vocabulary, in the order a message meets it.

  • Topic: a named log for one kind of event, such as orders or clicks.
  • Partition: one ordered list inside a topic. A topic with 3 partitions is 3 separate lists.
  • Key: a label on the message, such as ORD-1, used to choose the partition.
  • Offset: a message's position within its partition, starting at 0. Offsets belong to a partition, not to the whole topic.

Now 5 events through a topic with 3 partitions.

Input: 5 events, in the order they were produced.

ArrivalKeyEvent
1ORD-1created
2ORD-2created
3ORD-1paid
4ORD-3created
5ORD-2paid

To pick a lane we need a number for each key. Python gives every character a code with ord(). Add up the codes and take the remainder when dividing by 3.

For ORD-1 that is 79 + 82 + 68 + 45 + 49 = 323, and 323 divided by 3 leaves 2.

Sum for ORD-1: 79 + 82 + 68 + 45 + 49 = 323. 323 % 3 = 2.

Characterord() value
O79
R82
D68
-45
149

Only the last character differs, so the 3 keys land in three different lanes.

KeySum of codesSum % 3Partition
ORD-132322
ORD-232400
ORD-332511
5 events routed into 3 partitions. ORD-2 events go to partition 0, ORD-3 to partition 1, and both ORD-1 events to partition 2, each partition keeping its own offsets
5 events, routed by key. Each partition has its own offsets starting at 0, and each order's events stay together in the order they arrived.

Read the colours. Cyan is ORD-1, amber is ORD-2 and purple is ORD-3. Each colour stays inside 1 partition, and ORD-1's created event comes before its paid event at offsets 0 and 1.

Partitions 0 and 2 each hold 2 events and partition 1 holds one, so a consumer for each lane would be working in parallel.

A matching example

Here is the same thing in code. partition_for turns a key into a lane, and FakeKafkaTopic is a list of lists: one inner list per partition.

Python
def partition_for(key: str, n_partitions: int) -> int:
    return sum(ord(c) for c in key) % n_partitions

print(partition_for("ORD-1", 3))
print(partition_for("ORD-2", 3))
print(partition_for("ORD-3", 3))
2
0
1
Python
class FakeKafkaTopic:
    def __init__(self, name: str, n_partitions: int):
        self.name = name
        self.partitions = [[] for _ in range(n_partitions)]

    def produce(self, key: str, value: str) -> int:
        idx = partition_for(key, len(self.partitions))
        self.partitions[idx].append({"key": key, "value": value})
        return idx

topic = FakeKafkaTopic("orders", 3)
for key, value in [("ORD-1", "created"), ("ORD-2", "created"), ("ORD-1", "paid"),
                   ("ORD-3", "created"), ("ORD-2", "paid")]:
    topic.produce(key, value)

for i, part in enumerate(topic.partitions):
    print("partition", i, part)
partition 0 [{'key': 'ORD-2', 'value': 'created'}, {'key': 'ORD-2', 'value': 'paid'}]
partition 1 [{'key': 'ORD-3', 'value': 'created'}]
partition 2 [{'key': 'ORD-1', 'value': 'created'}, {'key': 'ORD-1', 'value': 'paid'}]

This matches the picture. Both ORD-1 events are in partition 2, with created first. A consumer reading partition 2 sees ORD-1's history in order and never has to wonder about the other orders.

A second, more realistic case: a ride-hailing app sends location pings for each driver. If the key is the driver_id, every ping from driver D-17 lands in 1 partition, in order, so a consumer can compute that driver's route without sorting.

If instead you picked no key at all, pings from one driver would spread across lanes and could be processed out of order.

Watch outDo not call hash(key) here

Python's built-in hash() for strings is randomised each time the interpreter starts. hash("ORD-1") would give a different lane after every restart, and the same key would stop mapping to the same partition. Use a stable function. The exercise grades the sum of character codes.

Watch outA popular key makes a hot partition

Partitions only share the work evenly if keys are spread evenly. If you used country as the key and most orders come from one country, 1 partition gets most of the traffic and its consumer falls behind while the others sit idle. Pick a key with many different values, like order_id.

How many lanes should a topic have?

Work backwards from the traffic. Say ShopKart expects 100 MB of events every second, and one reader can process about 10 MB a second.

Python
import math

target_mb_per_sec = 100
one_reader_mb_per_sec = 10

partitions = math.ceil(target_mb_per_sec / one_reader_mb_per_sec)
print(partitions)
10

10 lanes, so 10 readers can share the load. Going lower has a cost you will not see until it hurts.

  • Too few lanes: a topic with 4 lanes and 8 readers leaves 4 readers idle, because each lane is read by only one reader at a time.
  • Too many lanes: more files for the servers to manage, and slower recovery when one server fails.

What if one key is much bigger than the rest?

Say one big seller sends 70% of ShopKart's orders and you chose seller_id as the key. Every one of that seller's events lands in the same lane. That lane's reader falls behind while the other lanes sit nearly empty. This is a hot partition.

Two common fixes. Make the key finer, such as seller_id plus order_id, so the seller's orders spread over many lanes while each single order still stays in order. Or, when order does not matter, send the events without a key and let them spread evenly.

Common beginner questions

How many partitions should a topic have?

There is no single right number. More partitions allow more consumers to work at once, and also mean more to manage. A moderately busy topic often starts with somewhere around 6 to 12, but it depends on your traffic. Work out how many consumers you will eventually want.

Can I add partitions later?

Yes, though you cannot easily reduce them. Adding them changes the remainder, so keys can move. With our stand-in, ORD-1 gives 323 % 3 = 2 on 3 partitions but 323 % 4 = 3 on four. From that moment its new events go to a different lane than its old ones, which breaks the order guarantee across the change.

What if I need all messages in strict order?

Then you use 1 partition, and give up the parallel reading. Order is only guaranteed within a partition, so most designs only ask for order per key.

5 events, three lanes
  1. Producer: 5 events arrive: ORD-1 created, ORD-2 created, ORD-1 paid, ORD-3 created, ORD-2 paid.

    1. 5 events
    2. 3 lanes
  2. Producer: Each event carries its order id as the key. The key decides the lane, so nobody has to guess.

    1. key
    2. number
    3. lane
  3. Router: ORD-1 adds up to 323, and 323 divided by 3 leaves 2. ORD-1 goes to lane 2. ORD-2 goes to lane 0 and ORD-3 to lane 1.

    1. 323 % 3 = 2
    2. lane 2
  4. Lane 2: Both ORD-1 events sit in lane 2, created first and paid second, because a lane keeps its own order.

    1. offset 0: created
    2. offset 1: paid
  5. Consumers: Three readers, one per lane, work at the same time. None of them ever sees ORD-1 out of order.

    1. reader 0
    2. reader 1
    3. reader 2

What comes next

You know where messages live. The next lesson covers how a consumer remembers how far it has read in each partition. Those bookmarks are called offsets.

Practice

Run the sample, then open the exercise. Write partition_for(key, n) so that it returns the sum of the key's character codes, divided by n, keeping the remainder.

Call it with the key ORD-1 and 3 partitions, assign the result to result and print it. Check your answer against the worked arithmetic above.

Going further: retention, and the consumer that falls too far behind

A topic does not keep events forever. It keeps them for a retention period and then deletes the oldest. A consumer that stays within that window never notices. A consumer that falls behind by more than the window loses data it never read.

The log below keeps only the 5 newest events. A consumer last read up to offset 2 and was offline while 10 more events arrived. Predict where it can resume.

PythonRetention versus a slow consumer
RETENTION = 5                      # the log keeps only the newest 5 events

class Log:
    def __init__(self):
        self.start = 0             # offset of the oldest event still kept
        self.next = 0              # offset of the next event to be written
    def append(self, count):
        self.next += count
        self.start = max(self.start, self.next - RETENTION)

log = Log()
log.append(3)                      # offsets 0..2 exist
saved_position = 3                 # the consumer has read all of them
log.append(10)                     # the consumer is offline; 10 more events arrive

print("log keeps offsets", log.start, "to", log.next - 1)
if saved_position < log.start:
    print("saved position", saved_position, "is gone ->", log.start - saved_position, "events were deleted unread")
log keeps offsets 8 to 12
saved position 3 is gone -> 5 events were deleted unread

The log now keeps offsets 8 to 12, so the consumer's saved position of 3 no longer exists. The events 3 to 7 (5 events) were deleted before it read them. What happens next is a setting: the consumer may jump to the oldest kept event or to the newest, or stop with an error.

In every case there is a permanent gap.

The fix is operational. Set retention longer than the longest outage you want to survive, and alert on consumer lag so you act long before the lag approaches the retention window.

RetentionLag alertResult
7 daysAlert at 1 hour of lagTime to react, and a week of buffer
1 hourAlert at 30 minutes of lagAn outage over an hour loses data
7 daysNo lag alertWorks until the day nobody notices for 8 days

Try a new scenario

Retention is 24 hours and a weekend outage keeps your consumer down for 60 hours. What was lost, and what could you do now?

Work it out, then open the answer

The topic kept only the last 24 hours, so about 36 hours of events were deleted before the consumer could read them. If the source system still holds the data (a database, or files), re-extract that period from the source. If not, the gap is permanent. Longer retention and a lag alert would have prevented it.

Interview: how do you pick a partition key?

Pick the field whose order matters and that spreads load evenly, such as a customer or order id. A key with a few huge values creates a hot partition, and a random key loses the per-key order.

Close the lesson and explain it

Without scrolling up, explain what retention does and what happens to a consumer that falls behind by more than the retention period.

Practicals · load into the editor

After you read the theory, run these in the pane on the right. They execute in this tab, no cluster.

Finished this lesson?

Mark it as read when you could explain it to someone else.

Rate:
Was this useful?