Queue Safety with Idempotency Keys and Retry Contracts
Retries are mandatory in production systems. Duplicate side effects are optional. The difference is whether idempotency is explicit in your event contract.
1. Put idempotency keys in every command event
Do not generate keys in workers. Generate at command creation and pass through unchanged. The same key must survive retries and redelivery.
2. Store processing state with deterministic outcomes
Keep a dedup table with key, state, and response payload. If the same key returns, serve the stored outcome instead of running side effects again.
3. Separate transient failures from business failures
Transient errors should retry with backoff. Business rule failures should complete and record failure reason to avoid endless redelivery loops.
4. Define retry budget per queue class
High-value queue jobs get longer retry windows. Low-value jobs should fail fast and move to dead-letter queues for manual inspection.
5. Audit idempotency in tests
Add integration tests that intentionally replay the same event key and assert exactly-once side effects. If this test is missing, your queue safety is unverified.
Idempotency is not a utility function. It is a reliability contract between producers, consumers, and storage.