Ecosystem fit: Kafka Streams, Connect and Spring AMQP

Learning Objectives

By the end of this lesson you can:

• Name the three ecosystem layers (client libraries, integration frameworks, stream processing) and say which broker each layer assumes.
• Read an existing codebase and predict which broker it was written against, from imports and annotations alone.
• List the integration work a broker switch would force, and estimate whether that work is a sprint or a quarter.
• Match managed services (Confluent Cloud, MSK, Amazon MQ, CloudAMQP) to the ecosystem they carry with them.

Quick Review

Last lesson we established the operational shape of each broker: Kafka's cost sits in disk, partition count and rebalance mechanics, and it pays you back with replay and cheap retention. RabbitMQ's cost sits in queue count, connection churn and memory headroom, and it pays you back with per-message routing and flexible acknowledgements.

Keep that trade-off in mind, because ecosystem fit is the second axis of the same decision. A broker that is operationally fine but fights your codebase is still the wrong answer.

Theory: The Three Ecosystem Layers

When someone says "our stack is a Kafka shop" or "we're a RabbitMQ shop", they usually mean three separate things. Separate them and the decision gets much clearer.

flowchart TD
    App["🧩 Application need"] --> Lib["📚 Client library"]
    App --> Integ["🔌 Integration framework"]
    App --> Proc["🌊 Stream processing"]
    Lib --> K[("Kafka")]
    Lib --> R[("RabbitMQ")]
    Integ --> K
    Integ --> R
    Proc --> K

Layer 1, client libraries. These are near-symmetric. Kafka has first-class clients in every mainstream language; RabbitMQ speaks AMQP 0-9-1 plus the newer RabbitMQ Streams protocol. Neither broker blocks you here.

Layer 2, integration frameworks. This is where asymmetry starts. Kafka Connect is a platform: a worker process, a REST API, offset management, and hundreds of source and sink connectors maintained by vendors and the community. RabbitMQ has no direct equivalent. Its integration story is libraries plus plugins (Shovel, Federation, the management HTTP API) and connector support inside other tools. If your pipeline is mostly "move data from A to B without writing code", Kafka has a product for that and RabbitMQ has a pile of parts.

Layer 3, stream processing. Kafka Streams is a library that turns a Kafka topic into a stateful, fault-tolerant processor: joins, windowed aggregations, local RocksDB state stores, exactly-once via processing.guarantee=exactly_once_v2, and interactive queries against that state. There is no RabbitMQ counterpart of comparable scope. RabbitMQ Streams (the 3.9+ feature) gives you retained, replayable streams inside RabbitMQ, which is genuinely useful, but the processing framework around it is yours to build or to buy elsewhere.

On the Spring side the split is clean and worth memorising:

Spring Cloud Stream deserves a note: it puts a binder interface in front of both brokers, so the same functional bean can target either. That is real portability, but only for the subset of behaviour the binder exposes. It will not give you replay semantics on RabbitMQ or per-message routing on Kafka.

[!NOTE] A framework that supports both brokers is not the same as framework parity. The binder evens out the programming model, not the broker semantics underneath it. Always ask which features the binder silently drops.

Worked Examples

Example 1: The Spring AMQP codebase

You inherit a service with this shape:

@RabbitListener(queues = "orders.created")
public void onOrderCreated(OrderCreated event) { ... }

and a RabbitTemplate with a MessagePostProcessor that sets a per-message TTL for delayed retries.

Step 1, identify the layers. Spring AMQP is the client abstraction; TTL plus dead-letter exchange is a RabbitMQ-specific routing pattern.

Step 2, translate to Kafka. @KafkaListener replaces @RabbitListener, KafkaTemplate replaces RabbitTemplate. That part is an afternoon.

Step 3, find the forced work. Kafka has no native per-message delay. You would need a separate retry topic with a scheduled consumer, or a scheduler component. That is real design work, not a rename.

Verdict: switching costs you one sprint for the listener migration plus one for the retry redesign. If the only argument for Kafka is "the platform team runs Kafka", the retry redesign is where the argument usually dies.

Example 2: Postgres CDC into an event bus

Requirement: capture row changes from Postgres into downstream consumers, with replay for new consumers.

Step 1, look for a connector. Debezium on Kafka Connect is the well-trodden path: a source connector reads the WAL, Connect stores offsets in Kafka internal topics, and downstream consumers replay from a topic.

Step 2, price the RabbitMQ alternative. Debezium Server supports non-Kafka sinks, but the community material, examples and managed offerings are overwhelmingly Kafka-shaped. You would be writing custom glue and owning it.

Step 3, weigh replay. The requirement explicitly needs replay for new consumers, which is Kafka's home turf.

Verdict: strong Kafka signal, on both ecosystem and semantics.

Example 3: The stream-processing requirement

Requirement: a five-minute tumbling window of payment events, deduplicated by payment ID, queryable by an ops dashboard.

Step 1, check Layer 3. Kafka Streams gives you windowedBy, a state store keyed by payment ID, and interactive queries for the dashboard. Roughly 60 lines.

Step 2, price the RabbitMQ version. You consume from a queue, keep the window in Redis or Postgres, author your own checkpointing, and rebuild on restart.

Verdict: the ecosystem gap here is the largest of any layer. If stateful stream processing is core to the product, the broker decision is effectively already made.

Discussion and Reflection Prompts

• Take a service you own and list every import that names a broker. Which of the three layers do they sit in?
• Where has your team used Spring Cloud Stream's portability claim, and what broker-specific behaviour leaked through the binder anyway?
• Which of your pipelines are "move data from A to B" (connector-shaped) versus "transform and aggregate" (processor-shaped)?

Expert Corner

Three habits that save you from expensive surprises.

Grep before you argue. Run rg "spring-kafka|spring-amqp|kafka-clients|amqp-client" across the repos. The dependency list is a more honest statement of your architecture than any design doc.

Separate migration cost from redesign cost. Porting annotations is migration, and it is cheap. Replacing a routing pattern, a retry scheme or a stateful processor is redesign, and it is not. Quote them separately in the recommendation.

Check what the managed service carries. Confluent Cloud and Amazon MSK bring Kafka Connect and its connectors with them; MSK Connect runs connectors for you. Amazon MQ and CloudAMQP give you RabbitMQ with the management plugin, but no connector platform. The managed offering inherits the ecosystem asymmetry of the broker it hosts.

Common mistakes to avoid:

• Assuming Spring Cloud Stream makes brokers interchangeable. It makes the binder layer interchangeable.
• Forgetting that Kafka Streams exactly-once requires {{c2::exactly_once_v2}} and transactional producers, which raises latency and needs broker-side support.
• Treating RabbitMQ's lack of a stream processor as a RabbitMQ flaw. It is a scope choice. Pair it with a processor if you need one.
• Underestimating {{c3::Kafka Connect offset storage}}: connector state lives in Kafka internal topics, so retention settings on those topics become production-critical.

Quick recall check before the next lesson:

• The Spring project that talks to RabbitMQ is {{c1::Spring AMQP}}.
• Swapping a Spring Cloud Stream transport means changing the {{c4::binder}} dependency, not the business logic.

[
  { "front": "What is Spring AMQP?", "back": "The Spring project for RabbitMQ: RabbitTemplate, @RabbitListener, listener containers, spring-boot-starter-amqp." },
  { "front": "What is spring-kafka?", "back": "The Spring project for Kafka: KafkaTemplate, @KafkaListener, and Boot auto-configuration." },
  { "front": "What does Spring Cloud Stream abstract?", "back": "A binder interface over both brokers, so the same functional bean can target Kafka or RabbitMQ." },
  { "front": "What does Spring Cloud Stream NOT give you?", "back": "Broker parity. Replay, routing and delivery semantics still come from the broker underneath." },
  { "front": "What is Kafka Connect?", "back": "A worker platform with a REST API and hundreds of source and sink connectors for moving data without custom code." },
  { "front": "Where does Kafka Connect store connector offsets?", "back": "In Kafka internal topics, which makes their retention settings production-critical." },
  { "front": "What is Kafka Streams?", "back": "A library for stateful stream processing on Kafka: joins, windowing, local state stores, exactly-once." },
  { "front": "What setting enables exactly-once in Kafka Streams?", "back": "processing.guarantee=exactly_once_v2, which uses transactional producers." },
  { "front": "Does RabbitMQ have a Kafka Streams equivalent?", "back": "No. RabbitMQ Streams provides retained streams, but the processing framework is yours to build or buy." },
  { "front": "Which RabbitMQ features replace parts of Kafka Connect?", "back": "Shovel, Federation and the management HTTP API, plus connector support inside other tools." },
  { "front": "What is the best-supported Debezium deployment for CDC?", "back": "The Kafka Connect source connector; non-Kafka sinks exist but with less tooling and community material." },
  { "front": "How do you tell which broker a service targets, fast?", "back": "Grep the imports and dependencies for spring-kafka, spring-amqp, kafka-clients or amqp-client." },
  { "front": "Which managed Kafka services bring Connect with them?", "back": "Confluent Cloud managed connectors and Amazon MSK with MSK Connect." },
  { "front": "Which managed RabbitMQ services exist?", "back": "Amazon MQ and CloudAMQP, providing RabbitMQ with the management plugin but no connector platform." }
]

Summary and Next Steps

Ecosystem fit is not one question but three: which client library your code imports, whether you need a connector platform or just a library, and whether you need stateful stream processing. Kafka wins the second and third layers outright. RabbitMQ matches it on the first and offers routing patterns Kafka does not. {{c1::Spring AMQP}} and spring-kafka sit at the layer that is cheapest to change, and it is tempting to size the whole migration from that layer alone. Do not.

The useful output of this lesson is a two-column list: migration work versus redesign work. Next lesson, Worked decision walkthroughs: four use cases and a 2026 cost check, we take that list and the operational profile from the previous lesson, then price all four scenarios end to end.