Kafka V/S ZeroMQ V/S RabbitMQ: Your 15-Minute Architecture Guide

4 min readDec 6, 2020

In the previous post, I talked about Kafka and RabbitMQ on a high-level highlighting critical points where both messaging services complement each other.

Today, I am going to delve deep in core-functionalities and key performance indicators (KPI) such as scalability, distribution, throughput, durability and talk about which messaging service suffices for what type of test-case.

So, without further ado, let’s understand the key differences between Kafka, RabbitMQ and another popular messaging library ZeroMQ and infer insights.

Kafka: Empowering Real-Time Data Feeds

Quick Insight:

Initially built by LinkedIn and now maintained by Apache foundation, Kafka is a popular messaging bus for achieving low-latency, high-throughput and handling real-time data feeds. For a novice, you can think of Kafka as a big commit log which has stored records in the chronological order. The whole transaction is governed by Brokers, Producers and Consumers.

Use-Case:

If your use case revolves around high-throughput ingestion data streams, Kafka can be a worthy companion. Kafka is highly valuable for enterprise-level transactions, thanks to its massively scalable PUB/SUB queue architecture. Other popular use cases of Kafka include Message Broker, Log Aggregation and Activity Monitoring.

Kafka is available in a variety of bindings. Today we will be using Confluent-Kafka. Visit here to install Kafka on your machine.

Typical Kafka Producer:

from confluent_kafka import Producer

p = Producer({'bootstrap.servers': 'mybroker1,mybroker2'})

while True:
    p.poll(0)
    p.produce('mytopic', data.encode('utf-8'))

p.flush()

Typical Kafka Consumer:

from confluent_kafka import Consumer

c = Consumer({
    'bootstrap.servers': 'mybroker',
    'group.id': 'mygroup',
    'auto.offset.reset': 'earliest'
})

c.subscribe(['mytopic'])

while True:
    msg = c.poll(1.0)

    if msg is None:
        continue
    if msg.error():
        print("Consumer error: {}".format(msg.error()))
        continue

    print('Received message: {}'.format(msg.value().decode('utf-8')))

c.close()

Let’s use the Producer, Consumer and Data Generator to make the following transaction:

Data (Faker) -> Kafka Producer -> Kafka Topic -> Kafka Consumer -> Output

We will be sending the below JSON through Kafka, RabbitMQ and ZMQ. Alternatively, you can also use Python package, Faker. The package has a tremendous capability to generate a ton of consistent fake data.
{“Name”:”Milan Bhardwaj”,”Job”: “Development Engineer”, “Website”:”Medium”}

(Results have been accumulated at the end)

RabbitMQ: Streaming Text Oriented Messaging Protocol

Quick-Insight:

Developed by Pivotal Software, RabbitMQ is a popular open-source message broker that accepts and forwards binary blobs of data/messages/records. It is a pure implementation of a message broker leveraging AMQP (Advanced Message Queuing Protocol ), MQTT, and STOMP for communication amid services.

Use-Case:

RabbitMQ is favourable when you want the records or messages to deliver in the correct sequence. RabbitMQ features a superior messaging routing concept which is missing in Kafka. You can simply deploy a routing rule in RabbitMQ. RabbitMQ is a push-based broker unlike Kafka, which is always a pull-based design (polling for new messages on the server).

Visit here to install RabbitMQ on your machine.

Typical RabbitMQ Producer:

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(
    pika.ConnectionParameters(host='localhost'))
channel = connection.channel()

channel.queue_declare(queue='hello')
msg={"Name":"Milan Bhardwaj","Job": "Development Engineer", "Website":"Medium"}

while True:
    channel.basic_publish(exchange='', routing_key='hello', body=str(msg))

connection.close()

Typical RabbitMQ Consumer:

import pika
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
count=0

def callback(ch, method, properties, body):
    #body = your message

channel.basic_consume(queue="hello", on_message_callback=callback, auto_ack=True)
channel.start_consuming()

ZeroMQ: Power Zero Failure Distributed Tasks

ZeroMQ (ØMQ) is a concurrency framework backed messaging queue which has been developed by iMatix. ØMQ is very capable and highly efficient since it has solved the problem of distributed messaging. ØMQ operates on an API which looks very similar to classic raw TCP sockets. However, unlike a TCP socket establishing a direct one-to-one communication between two applications, ØMQ can handle several peer connections.

Popular ØMQ Design Patterns:

Synchronous Request/Response
Asynchronous Request/Response
Publish/Subscribe
Push/Pull
Exclusive Pair

We can say ØMQ is an abstraction of sockets. Although ØMQ has bindings in almost every programming language, but ØMQ sockets can talk to ØMQ sockets only.

Visit here to install ØMQ on your machine.

Typical ØMQ Producer:

import zmq
import time
import json
from faker import Faker

ctx = zmq.Context()
sock = ctx.socket(zmq.PUB)
sock.bind("tcp://*:1211")
msg={"Name":"Milan Bhardwaj","Job": "Development Engineer", "Website":"Medium"}

while True:
    sock.send_string(str(msg))

sock.close()
ctx.term()

Typical ØMQ Consumer:

import zmq

ctx = zmq.Context()
sock = ctx.socket(zmq.SUB)
sock.connect("tcp://127.0.0.1:1211")
sock.subscribe("") # Subscribe to all topics
print("Starting receiver loop ...")
count=0
while True:
    msg = sock.recv().decode("utf-8")
    #msg = your data

sock.close()
ctx.term()

Throughput Calculation:

Observed Speed: ZeroMQ (~1M m/s) > Kafka (~ 0.8M m/s) > RabbitMQ (~0.4M m/s)

Kafka is super fast when compared to RabbitMQ due to a maximized use of sequential disk reads and writes & Zero-Copy processing of messages.

ZeroMQ surpasses Kafka since it doesn’t support message persistence and often store messages only in limited buffers. On the other hand, Kafka goes to the disk and back.

(The test has been performed on a standard Intel® Core™ i3–5005U CPU @ 2.00GHz × 4 with 8 GB RAM)

The End Line:

Nevertheless, the trio Kafka, RabbitMQ and ZeroMQ are suitable to solve different problems. These should be distinguished and picked based on your cause.

Looking for durability? Go with Kafka. Looking for best speed without delivery guarantee? Go with ZMQ. Don’t want to miss either speed or persistence? RabbitMQ might be your companion.