Operations, scaling and retention in production

Picture the intern who ships the Kafka proof of concept and then disappears. A quarter later the topic hits 300 GB, nobody knows which consumer is lagging, and the one broker node they sized for the demo is running at 92% disk. This lesson is about the parts of a broker choice that show up on your pager at 3am rather than in the architecture diagram. Sizing, rebalancing, quorum, retention, monitoring: that is where the "real" cost lives.

Learning Objectives

By the end of this lesson you can:

• Size a Kafka cluster and a RabbitMQ cluster using the right unit for each (disk throughput and partitions versus queues, connections and memory).
• Explain why Kafka partition rebalancing is heavier than RabbitMQ consumer churn, and what to do about it.
• Choose Kafka quorum settings (replication.factor, min.insync.replicas, acks) and RabbitMQ queue mirroring or quorum queues with justification.
• Pick a retention policy per use case: time or size based on a log, acknowledgements or TTL based on a queue.
• Name the monitoring signals that actually predict incidents for each broker.

Quick Review

From the previous module: Kafka stores a partitioned log, so fan-out is free and filtering happens on the consumer side. RabbitMQ stores queues plus a routing graph of exchanges and bindings, so filtering happens broker-side and a message is gone once acknowledged. Keep those two pictures in mind, because almost every operational difference in this lesson falls out of them.

Theory

The unit of scaling is different

Kafka scales in partitions and bytes. A topic is a set of partitions spread across brokers. Partition count is the ceiling on parallelism: one partition is consumed by exactly one consumer in a group. Disk throughput per broker, not raw capacity, is usually the first limit you hit, because producers write sequentially and consumers read sequentially at line speed.

RabbitMQ scales in queues, connections and memory. Each queue is a single Erlang process, and each connection and channel costs memory. RabbitMQ is designed to keep queues short: messages are delivered and removed. Throughput scales by spreading load over queues and by sharding via consistent hashing exchanges or a sharded plugin, not by "adding partitions".

Worked Example 1: Sizing the same workload on each broker

Suppose you must move 4,000 messages per second, 2 KB each, retained for 7 days.

Payload rate: 4,000 × 2 KB = 8 MB/s ≈ 691 GB/day. Over 7 days of retention that is roughly 4.8 TB, before replication.

Kafka with replication.factor=3 needs about 14.5 TB of disk, and it needs it on fast local disks because brokers restore partitions by replaying log segments. A sane layout: 3 brokers, 1,000 partitions, about 100 partitions per broker, and node sizing driven by network and disk throughput rather than CPU.

RabbitMQ with 7-day retention is a different animal. If consumers keep up, the queues stay near empty and you need tens of GB, not terabytes. The sizing driver becomes connection count, channel count and the number of queues. If consumers fall behind, messages pile up in memory, page to disk, and throughput collapses. RabbitMQ does not want a 4.8 TB backlog; that is the wrong shape of problem for it.

Rule of thumb from that example: retention is cheap on Kafka and expensive on RabbitMQ. If your requirement contains "replay""or "keep for N days", you are describing a log.

Rebalancing and consumer churn

In Kafka, when a consumer joins or leaves a group, the group rebalances and partitions are reassigned. With the older eager protocol, every consumer stops, drops its partitions and resumes. That stop-the-world pause is invisible in a demo and very visible at 200 consumers with 2,000 partitions. Cooperative (incremental) rebalancing, controlled by partition.assignment.strategy, limits the pause to the moved partitions. Static group membership (group.instance.id) prevents a rolling redeploy from triggering a rebalance per pod.

In RabbitMQ, a consumer disconnecting affects only its channel and the queues it was attached to. There is no global reassignment, so a 50-pod deployment restarting slowly is not a cluster-wide event. The flip side: RabbitMQ has no built-in notion of "which consumer owns which partition", so ordering guarantees are per-queue, and your consumption order lives at the queue level with a single active consumer.

Quorum and durability settings

Kafka: replication.factor=3 plus min.insync.replicas=2 plus acks=all survives one broker loss with no data loss. acks=1 is faster and loses data when the leader dies. That last combination is the most common misconfiguration in production Kafka: a team sets acks=all and then a cluster-wide min.insync.replicas=1, and the guarantee quietly evaporates.

RabbitMQ: classic mirrored queues or, since 3.x, quorum queues (Raft-based) for durability. Quorum queues require an odd number of replicas (3 or 5), give clear leader election, and are the current recommendation for data you cannot lose. Classic mirrored queues are on the deprecation path; do not start new work on them.

Retention and cleanup policies

Kafka retention is a log property: retention.ms or retention.bytes per topic. Compaction (cleanup.policy=compact) keeps the latest value per key rather than expiring by time, which is what you want for a "current state" topic such as a customer profile or a config table. Compaction runs in the background, so a compacted topic's size is about keys, not messages.

RabbitMQ retention is per-queue: messages are removed when acknowledged, not after a timer. x-message-ttl and x-expires give you time-based expiry, and x-max-length plus a dead-letter exchange gives you overflow behaviour (drop-oldest, reject-newest, or dead-letter). There is no replay from the queue: once acknowledged, the message is gone. For replay you need an external store, or a Kafka-shaped log.

Worked Example 2: Choosing retention for three topics

1. Audit trail, must be replayable for 90 days. Kafka, retention.ms=7776000000, RF=3. Disk cost is real but predictable. RabbitMQ would need an external archive plus its own replay path.
2. Work queue for image transcoding, no replay needed. RabbitMQ. x-max-length of 100k, x-overflow=reject-publish to protect the broker, dead-letter on failure, TTL for stale jobs. A backlog of a few hours is normal.
3. "Current state" topic consumed by five services. Kafka with cleanup.policy=compact, min.cleanable.dirty.ratio left near default, and a delete.retention.ms for tombstones. Fan-out is free; each service reads the full log independently.

Monitoring: what to actually page on

Kafka: consumer group lag per partition (a single lagging partition means a hot key), under-replicated partitions (a broker is behind), offline partitions (emergency), disk usage trending past 70%, ISR shrink and expand events, request latency at p99, controller and broker GC pauses. Lag that grows steadily is a capacity problem; lag that spikes then recovers is a deploy.

RabbitMQ: queue depth and whether it is growing, messages_ready versus messages_unacknowledged (unacked growth means slow consumers), memory and disk alarms (broker blocks publishers, which is a hard stop, not a warning), connection and channel counts, file descriptor usage, and redelivery rate. A queue growing without bound at 3am is the classic RabbitMQ incident.

[!TIP] Page on symptoms your users feel (lag, queue depth, 5xx from the consumer), and dashboard everything else. Kafka gives you excellent per-partition telemetry; RabbitMQ gives you excellent per-queue telemetry. Neither one knows your business SLA.

Worked Example 3: Diagnosing the same symptom on both

Symptom: "processing is slow". On Kafka, check lag per partition. If one partition holds 90% of the lag, the key distribution is skewed and adding consumers will not help; you need more partitions or a better key. If all partitions are evenly lagging, add consumers up to the partition count, then add partitions.

On RabbitMQ, check messages_ready versus unacked. Ready high, unacked low: consumers are not keeping up, add consumers or reduce prefetch. Unacked high: the consumer is holding messages, usually a slow handler or a huge prefetch. Memory alarm firing: publishers are blocked, which is backpressure working as designed.

Discussion & Reflection Prompts

1. Your team wants 30-day retention on a 20k msg/s stream "just in case". What is the disk bill at RF=3 for 1 KB messages, and what question would you ask before agreeing?
2. You are deploying 40 consumer pods on Kafka and seeing a 20-second pause on every rollout. Which two settings do you reach for first?
3. A colleague says "RabbitMQ is easier to operate than Kafka". What is true in that statement, and where does it break down at scale?

Expert Corner

• Oversize partitions on day one. Increasing partitions later breaks key-to-partition ordering. Start with 2-4x your expected consumer count.
• Keep min.insync.replicas=2 and acks=all together, or do not claim durability. Audit it in a topic config diff, not in the docs.
• Quorum queues for anything you cannot lose. Classic mirrored queues have a known split-brain history; migrating later is a queue-by-queue operation with downtime.
• RabbitMQ loves short queues. If your design assumes a 10-million-message backlog, you have chosen Kafka and not realised it.
• Static membership plus cooperative rebalancing turns Kafka rollouts from an incident into a non-event.
• Compaction is not a database. It keeps the latest value per key eventually, not immediately; do not build a read-after-write state store on it without care.
• Track the ratio of operational cost to business value. Kafka clusters need more staff attention per node; RabbitMQ needs more attention per queue. Pick the shape that matches your team.

Summary & Next Steps

Kafka's operational cost is concentrated in disk, partition count and rebalance mechanics; RabbitMQ's is concentrated in queue count, connections and memory headroom. Kafka gives you replay and cheap retention by default. RabbitMQ gives you clean per-queue semantics, backpressure and simpler consumer churn, but it does not want to hold your archive. Choose durability settings aggressively, monitor backlog rather than CPU, and let "do I need replay for N days" drive the decision.

Key facts to lock in:

• {{c1::min.insync.replicas=2 with acks=all}} is the pairing that actually survives one broker loss in Kafka.
• Kafka retention is set per topic via {{c2::retention.ms or retention.bytes}}, while cleanup.policy=compact keeps {{c3::the latest value per key}} instead of expiring by age.
• RabbitMQ's recommended durable queue type in current versions is the {{c4::quorum queue}}, based on Raft with an odd number of replicas.
• In Kafka, {{c5::consumer group lag per partition}} is the signal that reveals a hot key, and under-replicated partitions indicate a broker falling behind.
• RabbitMQ protects itself with {{c6::memory and disk alarms}} that block publishers as backpressure, which is designed behaviour rather than a bug.

Next up: ecosystem fit. Once you have accepted the operational shape of each broker, the next question is what you get for free, Kafka Streams, Kafka Connect, or Spring AMQP and its listener container model.

[
  { "front": "What is the unit of scaling for Kafka?", "back": "Partitions and disk throughput; a partition is consumed by exactly one consumer in a group." },
  { "front": "What is the unit of scaling for RabbitMQ?", "back": "Queues, connections and memory; queues are meant to stay short and drain quickly." },
  { "front": "Which Kafka settings together guarantee no data loss on one broker failure?", "back": "replication.factor=3, min.insync.replicas=2 and acks=all." },
  { "front": "How much disk does a 4,000 msg/s, 2 KB, 7-day Kafka topic need at RF=3?", "back": "About 691 GB per day, roughly 4.8 TB retained, so about 14.5 TB after 3x replication." },
  { "front": "Which Kafka partition assignment mode limits rebalance pauses?", "back": "Cooperative or incremental rebalancing via partition.assignment.strategy." },
  { "front": "What does group.instance.id do in Kafka?", "back": "Static group membership, so a restarting pod does not trigger a rebalance." },
  { "front": "Which RabbitMQ queue type is recommended for durable messaging?", "back": "Quorum queues, Raft-based with an odd replica count (3 or 5)." },
  { "front": "How is retention configured on a Kafka topic?", "back": "Per topic with retention.ms or retention.bytes; cleanup.policy=compact keeps latest value per key." },
  { "front": "How does RabbitMQ remove messages?", "back": "On acknowledgement, plus optional x-message-ttl, x-expires and x-max-length with dead-lettering." },
  { "front": "Which RabbitMQ metric distinguishes slow consumers from missing consumers?", "back": "messages_unacknowledged high means slow handlers; messages_ready high means not enough consumers." },
  { "front": "Which Kafka metric reveals a hot key?", "back": "Consumer group lag concentrated on one partition." },
  { "front": "What happens when a RabbitMQ memory alarm fires?", "back": "Publishers are blocked, which is intentional backpressure, not a failure." },
  { "front": "Why is a large backlog a poor fit for RabbitMQ?", "back": "Messages page to disk and throughput drops; replay and long retention are Kafka territory." }
]

[
  { "term": "Under-replicated partitions", "definition": "Kafka partitions whose in-sync replica count is below the target, signalling a lagging broker." },
  { "term": "Quorum queue", "definition": "A RabbitMQ queue replicated with Raft, giving safe leader election and durable ordering." },
  { "term": "Log compaction", "definition": "Kafka cleanup policy that keeps the newest record per key instead of deleting by age." },
  { "term": "Prefetch", "definition": "RabbitMQ setting for how many unacknowledged messages a consumer may hold at once." }
]