1. The Dual-Write Hazard and the Outbox Pattern
Saving a record to your relational database and immediately publishing to Kafka in the same API route creates a race condition: if the Kafka broker is momentarily unreachable, your database has saved data that downstream consumers will never know about. We solve this by writing outbox records inside the primary database transaction, processed by Debezium CDC into Kafka.
2. Monotonic State Checks Over Naive Deduplication
Simple deduplication caches fail when events arrive out of order. We attach monotonically increasing version numbers to entity aggregates. Consumers discard or defer any event whose sequence number does not equal `entity.version + 1`.
Production Checklist & Takeaways
- Never publish directly to message brokers inside database transaction blocks.
- Treat idempotency keys as first-class entity attributes persisted in durable storage.
- Always equip dead letter queues with automated observability and alerting thresholds.