On this page
A payment app produces an event every time a customer pays. The fraud team wants those events instantly. The finance team wants them for reports. The notification service wants them to send a receipt. If the app sends each event directly to each of those systems, then every new team means new code in the app, and one slow system can slow down everyone. Apache Kafka solves this by sitting in the middle. The app writes each event once, to Kafka. Every other system reads it from there, at its own speed.
Think of Kafka as a very reliable notebook that many people can read from. The app writes new lines at the end. Readers keep their own bookmark. Reading a line does not erase it, so anyone can read it later, or read it again. The commands and code in this guide are typical examples, and they were not run here, because Kafka needs a running cluster.
Each event has a key, such as a customer id.
Order is kept inside each partition.
Same key, same partition.
Each keeps its own offset.
Reading deletes nothing.
The core words
- Event (or message). One thing that happened, with a key (for example a customer id), a value (the details) and a time.
- Topic. A named stream of events, such as
payments. Think of it as one notebook. - Producer. A program that writes events to a topic.
- Consumer. A program that reads events from a topic.
- Broker. A Kafka server. Several brokers together form a cluster, which gives Kafka its reliability. Recent versions no longer need a separate coordination service, because the brokers manage themselves, which makes a Kafka setup simpler than it used to be.
Partitions: splitting a topic
A busy topic cannot live on one machine, so Kafka divides each topic into partitions. A partition is one ordered list of events, and each event in it gets a number called an offset: 0, 1, 2, and so on.
Picture a payments topic with 3 partitions. Each partition is its own numbered list, and the lists grow independently:
| Partition | Offsets written so far |
|---|---|
| 0 | 0, 1, 2, 3 |
| 1 | 0, 1, 2 |
| 2 | 0, 1, 2, 3, 4 |
2 rules follow, and interviewers like both:
- Order is guaranteed inside 1 partition, not across the whole topic. Events in partition 0 are read in the order they were written. Between partitions, there is no promised order.
- Events with the same key go to the same partition. If you use the customer id as the key, all of Asha's payments land in 1 partition, so they stay in order, even though the topic as a whole is spread out. This is how you keep related events in sequence.
More partitions mean more parallel work, because different partitions can be handled by different consumers.
Offsets and retention: reading does not delete
A consumer remembers which offset it has read up to, and asks for the next one. Because Kafka does not delete an event when it is read, several things follow:
- A new consumer can start from the beginning and read the whole history.
- A consumer that was down can come back and carry on from its bookmark.
- Two teams can read the same topic without affecting each other.
Kafka does not keep events forever by default. It keeps them for a configured time, often 7 days, or up to a size limit, and then removes the oldest ones. This is called retention.
Consumer groups: sharing the work
A single consumer may not keep up with a busy topic. So consumers join a consumer group, and Kafka gives each partition in the topic to exactly one consumer in the group. The group splits the work.
Reads partitions 0 and 1.
Reads partition 2.
Reads all 3 partitions.
Has its own offsets.
Inside a group, each partition has one reader. Different groups do not affect each other.
The rules to remember:
- Each partition is read by one consumer in a group at a time. With 3 partitions and 2 consumers, one consumer reads 2 partitions and the other reads one.
- You cannot get more parallelism than partitions. With 3 partitions, a fourth consumer in the same group would have nothing to read and sit idle.
- If a consumer fails, its partitions are given to the others. This reshuffle is called a rebalance, and it is how the group keeps going.
- Different groups are independent. The fraud service and the reporting service each use their own group, so each gets every event, and each tracks its own offsets.
A new group needs to know where to start. The setting auto.offset.reset decides: earliest starts from the oldest event still kept, and latest starts from only new ones. It only applies when the group has no saved position yet.
Try it: typical commands
These are the basic commands that ship with Kafka. They create a topic with 3 partitions, write events to it, and read them back from the start.
# create a topic with 3 partitions
kafka-topics.sh --create --topic payments --partitions 3 --bootstrap-server localhost:9092
# type events, one per line (the producer)
kafka-console-producer.sh --topic payments --bootstrap-server localhost:9092
# read them all from the beginning (a consumer)
kafka-console-consumer.sh --topic payments --from-beginning --bootstrap-server localhost:9092In real systems the producers and consumers are programs in Python, Java or another language, but the ideas are the same.
Delivery and duplicates
Networks fail, so a message can occasionally be delivered more than once. The common safe habit is to assume this can happen and make the consumer idempotent, meaning that handling the same event twice gives the same result, as in what is a data pipeline. Kafka has features to reduce duplicates, but designing consumers to tolerate them is the dependable approach.
Kafka compared with a queue and a database
| A simple message queue | Kafka | A database | |
|---|---|---|---|
| Message after reading | Usually removed | Kept until retention ends | Kept |
| Many independent readers | Harder | Yes, each with its own offset | Yes, by querying |
| Replay history | No | Yes, within retention | Yes |
| Best for | Handing a job to one worker | Streams of events many systems share | Looking up current state |
Do you need Kafka?
Often not. If one nightly batch job is enough, a plain pipeline is simpler, as explained in batch vs stream processing. Kafka earns its place when events arrive continuously, several systems need them, and you want to replay them. The most important thing for a beginner is the picture above: topics, partitions, offsets and consumer groups. Practise them in the Streaming & Message Queues track, and read the Kafka interview questions for the usual follow-ups.