Cloud Pub/Sub and Amazon SQS operate under an at-least-once delivery model by default, meaning consumers will inevitably receive duplicate messages whenever acknowledgment deadlines expire before processing completes. Downstream consumers must be engineered for idempotency by tracking unique business or message identifiers so that redeliveries do not duplicate database rows or side effects. While features like ordering keys and regional exactly-once delivery exist, they introduce throughput trade-offs, and consumers must route repeatedly failing messages to dead-letter queues.
Why brokers trigger redelivery
Cloud message brokers track delivery progress using visibility timeouts and acknowledgment deadlines.
When a consumer worker pulls a message batch, the broker sets an acknowledgment deadline, such as thirty seconds. If the worker encounters a slow external database query or transient Garbage Collection pause and fails to acknowledge the message within thirty seconds, the broker assumes the worker died. The broker re-enqueues the message, delivering it to another worker while the initial worker may still be finishing its write.
Designing consumers for idempotent processing
Because duplicate deliveries are guaranteed over time, consumer applications must apply defensive patterns:
- Idempotent database writes: Use database primitives like INSERT ... ON CONFLICT DO UPDATE in PostgreSQL or MERGE statements in lakehouse tables, keyed on the message business ID. Repeated writes overwrite existing rows with identical values rather than creating duplicate entries.
- Distributed deduplication locks: Check a fast key-value store like Redis using atomic operations like SETNX on the event ID with a time-to-live matching the maximum redelivery window. If the key already exists, acknowledge and discard the message.
Advanced delivery features and dead-letter queues
Modern cloud brokers offer specialized configurations that alter standard consumer behavior:
- Ordering keys: In Cloud Pub/Sub and SQS FIFO, assigning an ordering key ensures messages sharing that key are delivered sequentially. However, ordering restricts parallel processing across partitions, reducing overall consumer throughput.
- Pub/Sub exactly-once delivery: Google Cloud Pub/Sub supports an exactly-once delivery setting on pull subscriptions within a single region. This feature prevents duplicates caused by acknowledgment deadline timeouts, but cross-region failovers or upstream publisher retries can still produce duplicate payloads.
- Dead-letter queues: When a poison pill message contains invalid syntax and crashes the consumer repeatedly, the broker redelivers it indefinitely unless configured with a maximum delivery count. Setting a dead-letter queue forwards failing messages to a separate quarantine topic after five attempts, unblocking the main pipeline while alerting on-call staff.