A dead letter topic is a secondary Kafka topic where corrupt, unparseable, or invalid poison messages are routed so they do not crash consumers or block partition consumption. When a consumer encounters a payload that fails deserialization or schema validation, it publishes the record to the dead letter topic along with contextual diagnostic headers before committing the original partition offset. This pattern prevents pipeline stalls, enables alerting on anomalous data volumes, and allows engineering teams to fix bugs and replay failed events later.
Why poison pills block partitions
In an ordered partition log, an unhandled deserialization error or null pointer exception halts the consumer thread. If the application crashes or retries indefinitely without advancing its offset:
- The consumer makes zero forward progress on that partition.
- Thousands of subsequent valid customer orders or payment events queue up behind the single bad record.
- Consumer lag mounts rapidly across the entire cluster.
Diverting unprocessable messages to a dead letter topic allows the consumer to commit the offending offset and continue processing healthy downstream traffic.
Enriching messages with diagnostic headers
When producing a record to a dead letter topic such as orders_dlq, write debugging metadata into Kafka record headers:
- x-original-topic: The source topic where the record originated.
- x-original-partition: The partition number of the failed record.
- x-original-offset: The offset of the poison message.
- x-exception-message: A truncated summary of the parsing error or validation failure.
- x-failure-timestamp: The epoch timestamp when the failure was caught.
Storing this metadata in headers keeps the original unparseable byte payload intact in the message value, preserving raw data for inspection.
Kafka Connect and replay workflows
Kafka Connect provides native dead letter queue support for sink connectors. Setting errors.tolerance = all, errors.deadletterqueue.topic.name = orders_dlq, and errors.deadletterqueue.context.headers.enable = true automatically routes bad rows without writing custom error code.
Set up Prometheus alerts to trigger whenever the message arrival rate on any dead letter topic climbs above zero. Once developers deploy a bug fix or adjust schema definitions, run a replay consumer utility that reads messages from orders_dlq and produces them back into the main orders topic.