Message Queue Evolution: IBM MQ, RabbitMQ, Kafka, and Pulsar

Message queue architecture evolved because applications kept asking for a different bottleneck to be removed. Early systems cared most about reliable delivery between a small number of business applications. Later systems cared about routing flexibility, then internet-scale throughput, and then cloud-native operations with cheaper long-term storage.

IBM MQ represents the classic transactional broker. Producers send a message to a broker-managed queue, and consumers take messages from that queue under strong control from the broker. The broker tracks acknowledgements, persistence, and recovery after a crash. This model fits banks and back-office systems because the operational question is usually, "Did this instruction arrive exactly once from the application point of view?" The tradeoff is that the broker is doing a lot of coordination work per message, which keeps semantics clear but limits horizontal scale and makes cross-datacentre operation expensive.

RabbitMQ kept the broker-centric model but made routing a first-class feature. Producers publish to an exchange, not directly to a queue. Bindings then decide which queues receive each message based on routing keys and exchange type. That is powerful when one event should feed several downstream workers, or when consumers need selective subscription patterns. RabbitMQ also works well for short-lived work queues where consumers acknowledge and discard messages quickly. Its main constraint is that it is still fundamentally optimised for broker-managed delivery rather than for a very large retained event log. When message history grows, replay becomes important, or throughput becomes dominated by partitioned disk writes, the model starts to strain.

Kafka changed the centre of gravity from broker-mediated delivery to an append-only distributed log. A topic is split into partitions, each partition is an ordered sequence of records, and consumers track their own offsets instead of having the broker delete a message after one successful read. That single design move changes several properties at once. Retention becomes natural because the log can keep data for hours, days, or months. Reprocessing becomes cheap because a consumer can rewind to an earlier offset. Throughput improves because brokers mostly append sequentially to disk and serve batches. The price is that Kafka asks clients and operators to think in partitions, consumer groups, lag, and rebalancing. Ordering is only guaranteed within a partition, not across a whole topic, and hot keys can produce uneven load.

Pulsar keeps the log-based model but separates the serving layer from the storage layer. Brokers handle client traffic, while BookKeeper stores ledgers durably. That separation makes it easier to add serving capacity without moving all stored data at the same time. It also supports tiered storage more naturally, so old segments can move to object storage while recent data stays on faster local media. In practice this means Pulsar tries to combine queue-style features, stream-style retention, and cloud-native elasticity in one platform. The tradeoff is added architectural complexity. Operators now manage brokers, bookies, metadata services, and storage policy together.

The important pattern is not that one system simply replaces the previous one. Each step reflects a different priority:

• IBM MQ prioritises durable transactional hand-off.
• RabbitMQ prioritises flexible routing and decoupled workers.
• Kafka prioritises retained event streams and very high throughput.
• Pulsar prioritises separation of compute and storage, multi-tenancy, and longer retention economics.

A team usually "evolves" only when its current pain changes. If you need strict queue semantics for payment instructions, Kafka is not automatically a better answer. If you need to replay clickstream events for new analytics jobs, a classic broker becomes awkward. If you need to scale storage and brokers independently across many tenants, a log system with a split architecture starts to look attractive.

So message queue evolution is really workload evolution. The architecture changes when the system stops being limited by delivery correctness alone and starts being limited by routing fan-out, replay, throughput, retention, or operational cost.