Messaging systems form the backbone of many critical applications today. An important question for software architects and engineers is the selection of the right messaging system that can address multiple requirements like cross-platform communication, fault-tolerance, latency needs, and scalability. Several open-source and commercial messaging systems are available today, and careful consideration is needed to select the right one that best suits the specific needs of the software application.

Messaging Systems In The Age of Cloud-Native & Microservices

Modern software components (e.g., microservices) communicate with each other in two forms: synchronous messaging, and asynchronous messaging. Asynchronous messaging, in which the production of messages and their processing/consumption are decoupled, is often the preferred approach, particularly for building highly-scalable and distributed systems. Furthermore, asynchronous messaging is generally deployed either through Message Queuing or through the Publish/Subscribe (Pub/Sub) communication pattern. Kafka and RabbitMQ are arguably the two most preferred open-source messaging systems today.

Kafka, originally released by LinkedIn in 2011, is a software bus framework implementation based on stream processing. It is essentially a distributed streaming platform that aims to provide high throughput, low latency message processing. Data is stored as topics, and each topic is associated with a partitioned log of messages. Messages are continuously appended to each partition as they arrive. Kafka follows a pull-based approach where consumers make message requests in batches. It is suitable for high-traffic applications like log aggregation and web activity tracking.

RabbitMQ, first released in 2007, is an implementation of a conventional message broker or service bus that initial supported Advanced Message Queuing Protocol (AMQP), but now also supports other protocols like MQ Telemetry Transport (MQTT) and STOMP (Streaming Text-Oriented Messaging Protocol). While its conventional use was in message queuing, particularly in SOA-based architectures, it also supports Pub/Sub (via message exchanges) today. Unlike Kafka, RabbitMQ follows a push-based approach, whereby messages are pushed to the consumers.

Apache Pulsar and NATS Streaming are two other popular platforms. Redis, an in-memory data store, is now increasingly also viewed as a viable option for messaging, particularly in use cases that involve short retention of messages. Other open-source and commercially-supported systems include ActiveMQ, Amazon SQS (Simple Queue Services), Azure Service Bus, NSQ, and so on.

Kafka versus RabbitMQ

Kafka achieves superior performance vis-à-vis RabbitMQ in the following aspects.

• Kafka can reportedly scale up to a million plus messages per second, while RabbitMQ generally scales better to lesser levels. In general, Kafka is considered to be better at horizontal scale-out, while RabbitMQ may be better at vertical scale-up.
• Message retention in Kafka is for a longer period of time while messages in RabbitMQ are generally evicted as soon as they get successfully consumed. This implies that Kafka consumers have the ability to access and consume messages from the past as often as needed.
• Unlike RabbitMQ, Kafka guarantees the ordered processing of messages, even for multiple consumers, without compromising on scalability.

On the other hand, RabbitMQ scores over Kafka in the following aspects.

• RabbitMQ is the preferred choice over Kafka for queuing-based applications. While Kafka is well suited for Pub-Sub, it is highly inefficient in conventional queuing use cases.
• Kafka is relatively more complex to implement and maintain than RabbitMQ, and often needs additional work. It is based on the dumb-broker/smart-consumer concept, while RabbitMQ is based on the dumb-consumer/smart-broker concept.
• RabbitMQ provides better mechanisms to handle failures in message processing (e.g., dead-letter exchanges) compared to Kafka.
• RabbitMQ provides support for the routing, filtering and timing of messages. For instance, it provides delayed messaging (e.g., in throttling scenarios), and the ability to configure the time-to-live attribute for each message. Kafka provides limited to no support on these capabilities.

Closing Comments

The past decade has witnessed the maturity of the conventional messaging systems, and the emergence of newer ones. Each has its own strengths and weaknesses. The key factors for selecting the messaging system include the available developer skills, data persistence or retention requirements, development & operational costs, ease of message routing & scheduling, fault-handling requirements, latency and scaling capacity of the messaging system. It is also not uncommon for companies to use more than one messaging system (e.g., both Kafka and RabbitMQ) in order to benefit from the advantages of both systems.