Exactly-once claims: Kafka transactions vs RabbitMQ patterns

Learning Objectives

By the end of this lesson you should be able to:

• Name the two hops where duplicates appear, and explain why "exactly-once" is a property of a read plus write pair, not of a network.
• Describe how Kafka's idempotent producer and transactions make a consume-transform-produce loop exactly-once.
• Explain why RabbitMQ solves the same problem with at-least-once delivery plus an idempotent consumer and a dedup store.
• Spot the exact boundary where Kafka transactions stop protecting you.
• Judge whether the cost of a Kafka transaction or the cost of a dedup table is the better trade for a given pipeline.

Quick Review

Last lesson we established that delivery semantics are not a switch on the broker. They are the position of one line of code relative to your business logic: Kafka's commitSync, RabbitMQ's basic.ack. Committing after processing gives you at-least-once.

That leaves one question, and it is the topic of this lesson: instead of tolerating duplicates, can we eliminate them? Both brokers say yes. They mean very different things by it.

Theory

Where duplicates actually come from

There are two hops in any pipeline, and each can duplicate independently:

1. The consume hop. You process a message, crash before committing, and the broker redelivers.
2. The produce hop. You send a result, the network times out, you retry, and the broker now holds the record twice.

Exactly-once means hop 1 and hop 2 succeed or fail as one unit. That is why the guarantee is always scoped to a specific pair of operations, never to "the broker" in general.

Kafka: idempotent producer plus transactions

Kafka attacks both hops separately.

Hop 2 is solved by the idempotent producer. With enable.idempotence=true, the producer gets a producer id and stamps every record with a per-partition sequence number. The broker rejects any sequence it has already written, so retries are free of duplicates.

Hop 1 needs more. A transaction wraps the output sends and the offset commit into one atomic unit:

producer.initTransactions();
try {
    producer.beginTransaction();
    for (var record : records) {
        producer.send(new ProducerRecord<>("output", record.key(), transform(record.value())));
    }
    producer.sendOffsetsToTransaction(offsets, consumerGroupMetadata);
    producer.commitTransaction();
} catch (Exception e) {
    producer.abortTransaction();
}

Downstream consumers set isolation.level=read_committed so they never see records from an aborted transaction. The commit becomes a single point of truth: either the outputs are visible and the offsets moved, or neither happened.

The boundary. Everything above lives inside Kafka. The moment your transform writes to Postgres, calls a payment API, or sends an email, you are outside the transaction and duplicates are back. This is the mistake I see most often: someone adds enable.idempotence=true, reads the docs headline, and declares the pipeline exactly-once while the Postgres insert still runs twice.

RabbitMQ: there is no transaction spanning both hops

RabbitMQ does have tx.select and tx.commit, but they batch a channel's publishes and acks, are synchronous and slow, and give you no atomicity with an external system. They are not the answer to this problem. Treat them as a legacy batching tool.

The realistic pattern is: at-least-once delivery plus idempotency.

• Ack only after processing (basic.ack after the business write).
• Give every message a stable identity: the broker's message_id, or better, a business key like order_id + event_type.
• On consume, attempt to insert that key into a processed_messages table with a unique constraint, in the same local transaction as the business write.
• If the insert conflicts, the message is a duplicate: ack it and move on.

The message may be delivered three times. The observable effect happens once. That is what people mean by "effectively once".

The trade, side by side

==Kafka gives you exactly-once inside Kafka; everywhere else you buy it with an idempotency key and a dedup table.==

Worked Examples

Example 1: a Kafka-to-Kafka enrichment

Orders land on orders.raw, a service enriches them and writes to orders.enriched.

Walk the crash points. Crash before commitTransaction: the transaction aborts, offsets are not committed, the consumer restarts from the old offset, and no enriched record is visible. Crash after commitTransaction but before the next poll: the offsets are already committed, so nothing is reprocessed. Both paths are safe. This is genuinely exactly-once, and it is cheap because there is no external system.

Example 2: the same service, but the sink is Postgres

Now the service writes to orders.enriched and inserts a row for a billing job. The Kafka transaction commits, the process crashes before the Postgres insert, and on restart the offsets say "done". The billing row is lost. Reverse the order (Postgres first, then commit) and a crash loses the Kafka record instead.

Fix: stop pretending the transaction covers Postgres. Make the insert idempotent with INSERT ... ON CONFLICT (event_id) DO NOTHING, where event_id is derived from the topic, partition and offset. Now a replay is harmless, and you can downgrade to at-least-once with a clear conscience.

Example 3: an order pipeline on RabbitMQ

A payment consumer reads from payments.q and writes to a ledger table. It crashes after the ledger insert but before basic.ack.

RabbitMQ redelivers. The consumer opens a local transaction, inserts processed_messages(msg_id = 'pay-8821'), gets a unique violation, and knows this is a replay. It acks without touching the ledger. Net effect: one ledger row, one payment. The dedup table is now the thing that needs monitoring, and its TTL must be longer than your worst-case redelivery window.

Example 4: choosing under cost pressure

You have a 40k msg/s Kafka-to-Kafka aggregation. Kafka transactions add coordinator overhead but remove an entire dedup subsystem. Take the transaction.

You have a 200 msg/s RabbitMQ order flow writing to a relational database that already has unique constraints on business keys. The unique index is your dedup store. Adding a transaction coordinator here would be over-engineering. Take the idempotent consumer.

You have a Kafka stream whose sink is a third-party HTTP API with no idempotency support. Neither option works cleanly. Best available: Kafka transaction for the internal hop, plus an outbox table and an idempotency key sent to the API if it supports one. If it does not, you have at-least-once and should say so out loud.

Discussion & Reflection Prompts

• Think of a pipeline you own. Which hop is currently unprotected, the consume side or the produce side?
• If your dedup table had no TTL and no index, how long before it became a production incident?
• Your team lead asks for "exactly-once delivery to the data warehouse". What question do you ask first?

Expert Corner

• Idempotence is not exactly-once. enable.idempotence=true protects one hop. Do not let the naming fool you in a design review.
• Put the dedup write in the same database transaction as the business write. Two separate writes give you the same race you were trying to remove.
• Choose the dedup key from business meaning, not from the broker. A broker message id changes on some republish paths; order_id + event_type does not.
• Fencing matters on Kafka. Two instances sharing a transactional.id will fence each other, which is what you want during failover and a nasty surprise during a misconfigured rolling deploy.
• Long transactions hurt. They hold back read_committed consumers and can hit transaction.timeout.ms. Keep the loop tight and flush early.
• Rule of thumb: if both ends are Kafka, use transactions. If either end is a database you control, use an idempotency key.

Summary & Next Steps

Kafka's exactly-once story is real and mechanical: idempotent producer for retries, transactions for atomic output plus offset commit, read_committed for readers. It stops at the Kafka boundary. RabbitMQ has no equivalent, so the community pattern is at-least-once with an idempotency key and a dedup store written inside your own transaction. The question is never "which broker is more exactly-once", it is "which cost can this pipeline afford".

Kafka's exactly-once guarantee is built from the {{c1::idempotent producer}} plus {{c2::transactions}}, and it only covers reads and writes that both live inside {{c3::Kafka}}.
Kafka consumers set `isolation.level` to {{c4::read_committed}} so they skip records from aborted transactions.
RabbitMQ reaches effectively-once by pairing at-least-once delivery with an {{c5::idempotency key}} and a deduplication store written in the same transaction as the business change.

[
  { "front": "Which two hops must be atomic for a true exactly-once pipeline?", "back": "The consume hop (processing once) and the produce hop (writing the result once)." },
  { "front": "Which Kafka setting deduplicates producer retries within a partition?", "back": "enable.idempotence=true, using a producer id and per-partition sequence numbers." },
  { "front": "What does a Kafka transaction add on top of an idempotent producer?", "back": "Atomic writes across partitions plus the consumer offset commit, all committed or aborted together." },
  { "front": "Which Kafka consumer setting hides records from aborted transactions?", "back": "isolation.level=read_committed." },
  { "front": "Where do Kafka transactions stop providing exactly-once?", "back": "At any side effect outside Kafka: a database write, an HTTP call, an email." },
  { "front": "Why is RabbitMQ's tx.select not a real exactly-once solution?", "back": "It only batches publishes and acks on one channel, is synchronous and slow, and gives no atomicity with an external system." },
  { "front": "What is the standard RabbitMQ route to effectively-once?", "back": "At-least-once delivery plus an idempotency key and a dedup store written with the business change." },
  { "front": "Where must the dedup record be written relative to the business write?", "back": "In the same local transaction, so both land or neither does." },
  { "front": "What does 'effectively once' mean compared with 'exactly once'?", "back": "The message may be delivered repeatedly, but the observable effect happens once because duplicates are rejected." },
  { "front": "What is the operational cost of the RabbitMQ dedup pattern?", "back": "A growing dedup table needing TTL and index maintenance, plus unique-constraint contention under load." },
  { "front": "Which broker makes exactly-once cheap for a Kafka-to-Kafka pipeline?", "back": "Kafka, because one transaction covers both the read and the write with no external coordinator." },
  { "front": "What is the common mistake after enabling idempotence in Kafka?", "back": "Assuming the whole pipeline is exactly-once when only the producer retry hop is protected." },
  { "front": "Why does the Kafka offset commit belong inside the transaction?", "back": "Otherwise a crash between processing and committing causes reprocessing or lost work." },
  { "front": "What is a good choice of dedup key?", "back": "A business key such as order_id plus event_type, not a broker-generated message id." }
]

Next up is the checkpoint for this module, where you will map each broker onto at-most-once, at-least-once and exactly-once from memory and defend the placement of the commit line for each.