Log vs queue: what each broker really stores

Grab a coffee. This is the lesson I wish someone had given me before I spent a week debugging a "lost message" incident that was not a lost message at all, just a queue doing exactly what queues do. Before we argue about delivery guarantees or throughput numbers, we need one mental model straight: Kafka and RabbitMQ do not store the same thing. Everything else in this course follows from that.

Learning Objectives

• Describe the append-only log model and explain what a Kafka partition actually contains.
• Describe the broker-managed queue model and explain when RabbitMQ removes a message.
• Name the three consequences of each model: replay, retention, and where consumer state lives.
• Predict, for a given scenario, whether a message can be read twice, and by whom.

Quick Review

You already know the queue primitive from any language runtime: producers enqueue, consumers dequeue, and a message is handled by exactly one worker. Hold on to one detail from that mental model, because it is the hinge of this lesson: in a classic queue, the act of consuming is what removes the item. Kafka breaks that assumption. RabbitMQ keeps it.

Theory

The queue model: RabbitMQ owns the state

In RabbitMQ the broker is a stateful router with a short memory. A producer publishes to an exchange, the exchange routes to one or more queues, and each queue holds messages waiting for a worker. The broker tracks which messages are in flight (delivered but not yet acknowledged) and which are ready. When a consumer sends an ack, ==the message is deleted from the queue==. That is the whole lifecycle.

The important part is who holds the truth. The broker holds it. Queue depth, unacked counts, redelivery flags: all broker-side. A consumer can crash without telling anyone, and RabbitMQ will requeue its unacked messages on its own.

The log model: Kafka owns the bytes, you own your position

Kafka's unit of storage is a partition, which is an append-only file of records. Every record gets a monotonically increasing number, the offset. Producers append. Nothing is ever edited in place, and nothing is removed when a consumer reads it. Records are deleted later by a retention policy, either time-based (retention.ms, commonly 7 days) or size-based.

Consumers do not remove anything. Each consumer group stores a committed offset, meaning "I have processed up to here", and that state lives on the client side (committed into the internal __consumer_offsets topic). Reset the offset, and the same records get delivered again.

flowchart LR
  subgraph kafka["Kafka: append-only log"]
    KP["✍️ Producer"] --> KT["📜 Partition log<br/>offsets 0..N"]
    KT --> KA["👓 Group A<br/>committed offset 42"]
    KT --> KB["👓 Group B<br/>committed offset 17"]
  end
  subgraph rabbit["RabbitMQ: broker-managed queue"]
    RP["✍️ Producer"] --> RX["🔀 Exchange"]
    RX --> RQ["📥 Queue"]
    RQ --> RW1["👷 Worker 1"]
    RQ --> RW2["👷 Worker 2"]
  end

The three consequences

[!NOTE] Replay is not a feature you switch on in Kafka. It is a property of the storage model. In RabbitMQ there is nothing to replay after ack, because the bytes are gone.

Worked Examples

Example 1: draining a RabbitMQ queue by hand. A queue holds 200,000 messages. Three competing consumers each handle about 500 msg/s, so aggregate throughput is 1,500 msg/s. Drain time is 200,000 / 1,500 ≈ 133 seconds, roughly 2 minutes 13 seconds. Now suppose worker 2 crashes having delivered but not acked 400 messages. Those 400 return to the ready state and jump back into the queue; the other two workers pick them up. Suppose instead that all three crash after acking 90,000 messages: those 90,000 are permanently gone, and only 110,000 remain. Same broker, two very different outcomes, decided entirely by the ack.

Example 2: Kafka retention disk math. A topic ingests 50 MB/s at peak, with replication factor 3 and 7 day retention. Per day: 50 × 86,400 = 4,320,000 MB ≈ 4.3 TB. Over 7 days that is about 30 TB of logical data, and with RF=3 the cluster holds roughly 91 TB on disk. That number does not shrink because consumers read fast, and it does not grow because consumers read slowly. Consumption is irrelevant to disk usage. Retention is the only lever.

Example 3: replay, and the wall at the end of it. A bad deploy on day 0 makes a consumer commit offsets past messages it never processed. You discover it 3 days later. Because retention is 7 days, the records still exist, and a reset (kafka-consumer-groups.sh --reset-offsets --to-datetime) replays them cleanly. Discover it 9 days later and the records are already deleted: the offset reset succeeds and silently reads nothing. ==The retention window is the real limit on replay.==

Example 4: fan-out cost. One topic carries 1,000,000 messages per day at 2 KB each, so 2 GB per day stored once. Five independent services each need every message. In Kafka that is five consumer groups over the same 2 GB. In RabbitMQ that is five queues bound to a fan-out exchange, roughly 10 GB per day, and each queue drains independently.

Discussion & Reflection Prompts

• Think of a system you have built recently. If a downstream service needed to reprocess yesterday's data, could it, or would you be reconstructing state from logs?
• Which model matches your team's operational temperament: a broker that remembers exactly what is in flight, or a broker that remembers nothing and lets clients track position?
• Where in your current architecture does a message being "consumed once" hide a second consumer that will want it later?

Expert Corner

• Common mistake: assuming a RabbitMQ queue is a small database. It is a buffer. A queue with 5 million messages is usually a symptom (a consumer is dead), not a design.
• Rule of thumb: if the question "can I replay this?" has no clear yes, you probably want a log.
• Rule of thumb: if the question "does someone need to see this exactly once, right now, and then never again?" is the whole story, a queue is simpler and cheaper.
• RabbitMQ does ship streams (since 3.9), a log-like abstraction. It is genuinely useful, but the default mental model for the vast majority of deployments is still the queue. Do not build your architecture on the exception.
• Quorum queues replicate for durability, but replication changes failure behavior, not the deletion model. Ack still means gone.

Summary & Next Steps

Kafka stores an append-only log and lets consumers hold their own position, which buys replay, cheap fan-out, and consumption-independent retention. RabbitMQ stores messages in a broker-managed queue and deletes on ack, which buys per-message routing control, broker-side redelivery, and a much simpler mental model. Neither is better in the abstract; they answer different questions.

Check your recall:

In Kafka, a message is not removed when it is consumed; it is removed when the topic's {{c1::retention}} policy expires it.
A consumer group stores its read position, called an {{c2::offset}}, on the client side rather than in the message store.
In RabbitMQ, a message disappears once the consumer sends an {{c3::ack}}.
Replay is cheap in Kafka because a consumer can reset its committed offset to an earlier position, as long as the data is still inside the {{c4::retention window}}.
In RabbitMQ, giving a second independent consumer every message requires a second {{c5::queue}} and a second copy of the data.

[
  { "front": "What does a Kafka partition really store?", "back": "An append-only, immutable log of records, each identified by a monotonically increasing offset." },
  { "front": "What determines when Kafka deletes a record?", "back": "The topic retention policy (time or size), not whether anyone consumed it." },
  { "front": "Where does Kafka keep consumer progress?", "back": "Client side, as committed offsets per consumer group, not inside the stored messages." },
  { "front": "What happens to a RabbitMQ message after a successful ack?", "back": "The broker deletes it from the queue." },
  { "front": "Who holds consumer state in RabbitMQ?", "back": "The broker, via the unacked message set and delivery tags." },
  { "front": "Can you replay a RabbitMQ message after it has been acked?", "back": "No, the message no longer exists." },
  { "front": "Cheapest way to fan out one Kafka topic to five independent services?", "back": "Five consumer groups on the same topic: one stored copy of the data." },
  { "front": "How do you give five independent consumers every message in RabbitMQ?", "back": "Five queues bound to a fan-out exchange: five copies of the data." },
  { "front": "What happens to unacked messages when a RabbitMQ consumer crashes?", "back": "They are requeued and redelivered, with the redelivered flag set." },
  { "front": "What is Kafka's ordering guarantee at the storage level?", "back": "Ordering within a single partition only, not across the topic." },
  { "front": "What limits how far back a Kafka consumer can replay?", "back": "The retention window; records older than that have already been deleted." },
  { "front": "What are RabbitMQ streams?", "back": "An optional log-like abstraction in RabbitMQ, not the default queue model that most deployments use." },
  { "front": "Why does consumer speed not affect Kafka disk usage?", "back": "Because retention, not consumption, is what deletes records." }
]

Next up: at-most-once and at-least-once, worked through both brokers. Now that you know what each one stores, we can trace exactly where a message can be lost or duplicated in each pipeline.