RabbitMQ

There are dozens of message brokers: RabbitMQ, ActiveMQ, ZeroMQ, Apache Kafka, GCP Pub/Sub, Amazon SQS, Microsoft Azure Service Bus, and the list goes on and on. 

Use need message brokers when have a large volume of data, and want system to be able to cope with sudden fluctuations in traffic. However, using a Message Queue provides other benefits. One such benefit is separation of concerns.

Process A can perform some task and put a message onto a Queue.
Process B can read the message from that Queue and perform a task.
However, Process A has absolutely no dependency or even knowledge that Process B exists and vice versa.
In fact, we could replace Process A with another process, or several processes, without affecting Process B in the slightest.
The second advantage is resilience. Let’s imagine that Process B fails: if Process A were calling Process B, then that would cause both processes to fail; however, since all Process A is doing is writing to a queue, the messages will simply remain in the queue until such time as they are picked up.

RabbitMQ allows two/or more clients to communicate (decouple), these clients are online in different times.

RabbitMQ using AMQP protocol

In the same TCP connection we can have many channels

In RabbitMQ it is possible to subscribe only to a subset of the messages.

Concepts

Exchanges

Messages are not published directly to a queue; instead, the producer sends messages to an exchange. An exchange is responsible for routing the messages to different queues with the help of bindings and routing keys.

Binding

A binding is a link between a queue and an exchange.

Routing key

The routing key is a message attribute the exchange looks at when deciding how to route the message to queues (depending on exchange type).

Exchanges, connections, and queues can be configured with parameters such as durable, temporary, and auto delete upon creation. 

Durable exchanges: survive server restarts and last until they are explicitly deleted. 

Temporary exchanges: exist until RabbitMQ is shut down. 

Auto-deleted exchanges: are removed once the last bound object is unbound from the exchange.

In RabbitMQ, there are four different types of exchanges that route the message differently using different parameters and bindings setups. Clients can create their own exchanges or use the predefined default exchanges which are created when the server starts for the first time.

Types of Exchanges

1) Direct: The message is routed to the queues whose binding key exactly matches the routing key of the message. For example, if the queue is bound to the exchange with the binding key pdfprocess, a message published to the exchange with a routing key pdfprocess is routed to that queue.

2) Fanout: A fanout exchange routes messages to all of the queues bound to it.

3) Topic: The topic exchange does a wildcard match between the routing key and the routing pattern specified in the binding.

4) Headers: Headers exchanges use the message header attributes for routing.

Publish and subscribe messages

RabbitMQ uses a protocol called AMQP by default. To be able to communicate with RabbitMQ you need a library that understands the same protocol as RabbitMQ. Download the client library for the programming language that you intend to use for your applications.

Steps to publish a message/consuming a message:

1) Set up/create a connection object. The username, password, connection URL, port, etc., will need to be specified. A TCP connection will be set up between the application and RabbitMQ when the start method is called.

2) Create a channel in the TCP connection, then the connection interface can be used to open a channel through which to send and receive messages.

3) Declare/create a queue. Declaring a queue will cause it to be created if it does not already exist. All queues need to be declared before they can be used.

Set up exchanges and bind a queue to an exchange in subscriber/consumer. All exchanges must be declared before they can be used. An exchange accepts messages from a producer application and routes them to message queues. For messages to be routed to queues, queues must be bound to an exchange.

In publisher: Publish a message to an exchange

In subscriber/consumer: Consume a message from a queue.

4) Close the channel and the connection.

When to use RabbitMQ over Kafka? 

RabbitMQ is a solid, general-purpose message broker that supports several protocols such as AMQP, MQTT, STOMP, etc. It can handle high throughput. A common use case for RabbitMQ is to handle background jobs or long-running task, such as file scanning, image scaling or PDF conversion. RabbitMQ is also used between microservices, where it serves as a means of communicating between applications, avoiding bottlenecks passing messages.

Kafka is a message bus optimized for high-throughput ingestion data streams and replay. Use Kafka when you have the need to move a large amount of data, process data in real-time or analyze data over a time period. In other words, where data need to be collected, stored, and handled. An example is when you want to track user activity on a webshop and generate suggested items to buy. Another example is data analysis for tracking, ingestion, logging or security.

Kafka can be seen as a durable message broker where applications can process and re-process streamed data on disk. Kafka has a very simple routing approach. RabbitMQ has better options if you need to route your messages in complex ways to your consumers. Use Kafka if you need to support batch consumers that could be offline or consumers that want messages at low latency. 

In order to understand how to read data from Kafka, we first need to understand its consumers and consumer groups. Partitions allow you to parallelize a topic by splitting the data across multiple nodes. Each record in a partition is assigned and identified by its unique offset. This offset points to the record in a partition. In the latest version of Kafka, Kafka maintains a numerical offset for each record in a partition. A consumer in Kafka can either automatically commit offsets periodically, or it can choose to control this committed position manually. RabbitMQ will keep all states about consumed/acknowledged/unacknowledged messages. I find Kafka more complex to understand than the case of RabbitMQ, where the message is simply removed from the queue once it's acked.

RabbitMQ's queues are fastest when they're empty, while Kafka retains large amounts of data with very little overhead - Kafka is designed for holding and distributing large volumes of messages. (If you plan to have very long queues in RabbitMQ you could have a look at lazy queues.)

Kafka is built from the ground up with horizontal scaling (scale by adding more machines) in mind, while RabbitMQ is mostly designed for vertical scaling (scale by adding more power).

RabbitMQ has a built-in user-friendly interface that lets you monitor and handle your RabbitMQ server from a web browser. Among other things, queues, connections, channels, exchanges, users and user permissions can be handled - created, deleted and listed in the browser and you can monitor message rates and send/receive messages manually. Kafka has a number of open-source tools, and also some commercial once, offering the administration and monitoring functionalities. I would say that it's easier/gets faster to get a good understanding of RabbitMQ.

In general, if you want a simple/traditional pub-sub message broker, the obvious choice is RabbitMQ, as it will most probably scale more than you will ever need it to scale. I would have chosen RabbitMQ if my requirements were simple enough to deal with system communication through channels/queues, and where retention and streaming is not a requirement.

There are two main situations where I would choose RabbitMQ; For long-running tasks, when I need to run reliable background jobs. And for communication and integration within, and between applications, i.e as middleman between microservices; where a system simply needs to notify another part of the system to start to work on a task, like ordering handling in a webshop (order placed, update order status, send order, payment, etc.).

In general, if you want a framework for storing, reading (re-reading), and analyzing streaming data, use Apache Kafka. It’s ideal for systems that are audited or those that need to store messages permanently. These can also be broken down into two main use cases for analyzing data (tracking, ingestion, logging, security etc.) or real-time processing.

More reading, use cases and some comparison data can be found here: https://www.cloudamqp.com/blog/2019-12-12-when-to-use-rabbitmq-or-apache-kafka.html

	RabbitMQ	Apache Kafka
What it is?	RabbitMQ is a solid, mature, general purpose message broker	Apache Kafka is a message bus optimized for high-ingress data streams and replay
Primary use	Message queue for communication and integration within, and between applications. For long-running tasks, or when you need to run reliable background jobs.	A framework for storing, reading (re-reading), and analyzing streaming data. Optimal for
License	Open Source: Mozilla Public License	Open Source: Apache License 2.0
Written in	Erlang	Scala (JVM)
First Version Released	2007	2011
Persistence	Persist messages until they are dropped on the acknowledgement of receipt	Persists messages with an option to delete after a retention period
Replay	No	Yes
Routing	Supports flexible routing which can return information to a consumer node	Does not support flexible routing, must be done through separate topics
Message Priority	Supported	Not supported
Monitoring	Available through a built-in UI	Available through third-party tools such as when deployed on CloudKarafka or through Confluent
Language Support	Most languages are supported	Most languages are supported
Secure Authentication	Supports standard authentication and OAuth2	Supports Kerberos, OAuth2, and standard authentication