ABC (Accelerator, Break and Clutch) of MQ

Smoke signals, carrier pigeons, courier companies, emails – we as human beings have always revolutionised our ways to communicate, irrespective of distances between us. The same need exists in the computer world too – a method to send, receive, and create the message itself. This is how MQ (Message Queue) came into being.

Just like humans, some demand quick answers, such as when you eagerly tap someone on the shoulder or call them by their name from far away, and sometimes you send an email and wait for the answer. Sometimes you do not expect an answer at all.
Software has the same need for communication. There are different ways to communicate, even for IT systems. One of these ways is to connect two systems with each other via a queue.
Caution : This article is going to be a little longer than usual, cause I want to offer 100% coverage of MQ. I can guarantee that after reading this whole article you can call yourself an MQ Expert (bit exaggeration, but you got the point right) .

What is a message queue?
MQ is a component for inter-process communication that passes control, content, queries or messages to another component / service / module.
To illustrate message queues let me give you a metaphorical example, think of it as ordering at your favorite restaurant. A front-line worker takes your order for your favorite meal. The front-line worker then makes a request to the kitchen for your food. The kitchen receives the order and makes the food. In this scenario, an instruction was received and action was performed that triggered an event – order taken, request triggered, an action performed.
While the kitchen is making your food, the front-line worker doesn’t wait until it appears, they keep taking orders from other customers. The kitchen continues to operate and process through orders. Each point has its own responsibilities and doesn’t wait for the other to finish what they’re doing, and neither is constrained by time. That’s it. Simple right?
If you are a developer then you’re good to go ahead, if you are not then you can skip the coding part and jump to the section that highlights the real time applications and use cases of MQ.

Deep dive into MQ :
1.MQ workflow diagram :

Don’t worry if you don’t understand the heavy word such as BROKER. I am going to cover them. Right now I want you to be familiar with the process, which says there is someone who is sending a message, we will call them PRODUCER. The person on the other hand who is receiving the message is CONSUMER. BROKER is the middle man who does the heavy lifting for us.

2.Installing MQ Broker :

Here I’m using RabbitMQ as my message broker software. There are various brokers available such as Apache ActiveMQ, AWS MQ, Kafka (Kafka is slightly different from MQs – that alone deserves a separate article!) etc. There are solid reasons for me to select RabbitMQ.

• It provides graphical management UI (ActiveMQ doesn’t, unless you are skilled enough to do it through third party integration tool SingalFX).
• RabbitMQ supports various protocols such as AMQP and STOMP.
• It supports various exchanges such as fanout, direct and topic (in simple words if i’ve to tell you the difference between them, then think it as WhatsApp uses fanout exchange, WiFi or Bluetooth handshakes uses direct exchange and two microservices communicate using topic exchange, I said it with simplification so your mind don’t get explode).
• It supports clustering (I won’t tell you about clustering, cause it will blow your mind).
• It is as fast and agile as a Rabbit 🙂
• And many more.

2.1 steps to install RabbitMQ :

Linux : https://www.youtube.com/watch?v=eazz-Je4HAA&t=148s

Windows : https://www.youtube.com/watch?v=V9DWKbalbWQ

Sorry! I wish I could write the steps, but the article is already long enough.

After successful installation you will see the screen like this :

3.Codding :
Things are getting serious now, we’re gonna do the hands on the coding part. 

You can also find the code on my github : https://github.com/viveksoni100/MQDemo

3.1 application.properties file :

server.port=9292

spring.rabbitmq.host=localhost

spring.rabbitmq.port=5672

spring.rabbitmq.username=vivek

spring.rabbitmq.password=Vivek@123

3.2 pom.xml file :

<dependency>

            <groupId>org.springframework.boot</groupId>

            <artifactId>spring-boot-starter-amqp</artifactId>

        </dependency>

        <dependency>

            <groupId>org.springframework.amqp</groupId>

            <artifactId>spring-rabbit-test</artifactId>

            <scope>test</scope>

</dependency>

3.3 MQConfig.java file :


package com.mq.demo.config;

import org.springframework.amqp.core.*;
import org.springframework.amqp.rabbit.connection.ConnectionFactory;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.amqp.support.converter.Jackson2JsonMessageConverter;
import org.springframework.amqp.support.converter.MessageConverter;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

/**
* @author viveksoni100
*/
@Configuration
public class MQConfig {

public static final String QUEUE = “vaccine_queue”;
public static final String EXCHANGE = “vaccine_exchange”;

// creating queue
@Bean
public Queue vaccineQueue() {
return new Queue(QUEUE);
}

// creating exchange of type topic
@Bean
public TopicExchange exchange() {
return new TopicExchange(EXCHANGE);
}

// binding our queue with exchange (method of message transfer)
@Bean
public Binding binding(TopicExchange exchange) {
return BindingBuilder.bind(vaccineQueue()).to(exchange).with(vaccineQueue().getName());
}

// utility for doing essential conversions
@Bean
public MessageConverter converter() {
return new Jackson2JsonMessageConverter();
}

// making the connection with RabbitMQ server
@Bean
public AmqpTemplate template(ConnectionFactory connectionFactory) {
final RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
rabbitTemplate.setMessageConverter(converter());
return rabbitTemplate;
}
}

3.3 Citizen.java file :

package com.mq.demo.dto;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
* @author viveksoni100
*/
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Citizen {

private String name;
private String adharID;

}

3.4 Vaccine.java file :

package com.mq.demo.dto;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
* @author viveksoni100
*/
@Data
@AllArgsConstructor
@NoArgsConstructor
public class Vaccine {

private String name;
private float dose;

}

3.5 VaccinatedCitizen.java file :

package com.mq.demo.dto;

import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;

/**
* @author viveksoni100
*/
@Data
@AllArgsConstructor
@NoArgsConstructor
public class VaccinatedCitizen {

private Citizen citizen;
private Vaccine vaccine;
}

3.6 VaccineDrive.java file :

package com.mq.demo.publisher;

import com.mq.demo.config.MQConfig;
import com.mq.demo.dto.Citizen;
import com.mq.demo.dto.VaccinatedCitizen;
import com.mq.demo.dto.Vaccine;
import org.springframework.amqp.core.Queue;
import org.springframework.amqp.rabbit.core.RabbitTemplate;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.UUID;

/**
* @author viveksoni100
*/
@RestController
public class VaccineDrive {

@Autowired
private RabbitTemplate template;

@Autowired
@Qualifier(“vaccineQueue”)
private Queue VaccineQueue;

@GetMapping(“vaccinateCitizen”)
public String vaccinateCitizen() {

Citizen citizen = new Citizen();
citizen.setName(“Vivek Soni”);
citizen.setAdharID(UUID.randomUUID().toString());

Vaccine vaccine = new Vaccine();
vaccine.setName(“Pfizer”);
vaccine.setDose(1.5f);

VaccinatedCitizen vaccinatedCitizen = new VaccinatedCitizen();
vaccinatedCitizen.setCitizen(citizen);
vaccinatedCitizen.setVaccine(vaccine);

// sending message to our queue
template.convertAndSend(MQConfig.EXCHANGE, VaccineQueue.getName(), vaccinatedCitizen);

return “Success!”;

}
}

3.7 VaccineQueueListener.java file :

package com.mq.demo.consumer;

import com.mq.demo.config.MQConfig;
import com.mq.demo.dto.VaccinatedCitizen;
import org.springframework.amqp.rabbit.annotation.RabbitListener;
import org.springframework.stereotype.Component;

/**
* @author viveksoni100
*/
@Component
public class VaccineQueueListener {

@RabbitListener(queues = MQConfig.QUEUE)
public void consumeMessageFromQueue(VaccinatedCitizen vaccinatedCitizen) {
System.out.println(“Congratulations ” + vaccinatedCitizen.getCitizen().getName() +
” you are jabbed with + ” + vaccinatedCitizen.getVaccine().getName() + ” successfully!”);
}
}

Congratulations, you have successfully configured your queue with MQ server. Now run your program and check the queue, exchange tabs by logging in RabbitMQ server (localhost:15672). If you are able to see vaccine_queue and vaccine_exchange created then hurray! Our task is 99% completed. Now let’s test our program.

4.Execution :

Hit our REST API in the browser : http://localhost:9292/vaccinateCitizen

In the program console we should receive the message sent by our service :

5. Application / use cases / success story of MQs :

Now you might think, “where on earth would a message queue fit in my architecture”? The simple and quick answer is when: 

• you have “timeout errors” due to too many requests at the same time.
• you need a decoupled way to communicate between or within your application.
• you are polling a data store too often and you want this data store to be available to answer qualified queries instead
• you need to scale up and down during peak hours

In other words, any time a task is not part of a basic user transaction, and/or the results don’t impact a user response, that’s a perfect job for a message queue.

Long-running processes and background jobs

When requests take a significant amount of time, it is the perfect scenario to incorporate a message queue.

Imagine a web service that handles multiple requests per second and cannot under any circumstances lose one. Plus the requests are handled through time-consuming processes, but the system cannot afford to be bogged down. Some real-life examples could include:

• Images Scaling
• Sending large/many emails
• Search engine indexing
• File scanning
• Video encoding
• Delivering notifications
• PDF processing
• Calculations

The middleman in between microservices

For communication and integration within and between applications, i.e. as the middleman between microservices, a message queue is also useful. Think of a system that needs to notify another part of the system to start to work on a task or when there are a lot of requests coming in at the same time, as in the following scenarios:

• Order handling (Order placed, update order status, send an order, payment, etc.)
• Food delivery service (Place an order, prepare an order, deliver food)
• Any web service that needs to handle multiple requests

Hemnet (success story – 1) :

Hemnet is a property listing platform that moved from a fully on-premise solution to a new cloud-based solution in less than a year. The entire story of Hemnet is available in another blog that goes into detail on how RabbitMQ has been deployed as a pipeline for image scaling, among other things. 

Parkster (success story – 2) :

One of the fastest growing digital parking services, Parkster, has broken down its monolithic architecture into faster, decoupled microservices. Like Netflix, Parkster initially began life as a monolith to prove its business model. They quickly realized that this type of system – where the entire application is built into a single unit, with one codebase and one system – isn’t efficient or effective.

A few years into building the company, Parkster began breaking up the monolith into multiple small microservices communicating via message queues.

Softonic (success story – 3) :

Softonic, the software and app discovery portal, has more than 100 million users every month and delivers over 2 million downloads every single day. The portal has a constant flow of events and commands between the services it offers, and RabbitMQ is the message queue of choice, contributing to the fast, and effective architecture Softonic is known for. 

FarmBot (success story – 4) :

FarmBot is an open-source robotic hardware kit that enables interaction with agricultural projects in an efficient, fun way. FarmBot employs physical sensors and actuators that require a method of communication between the physical garden and the software. 

The message queue used is RabbitMQ, a real-time message broker, so there is no need to check for new messages. Messages such as a user clicking the “move” button on the interface are sent back and forth between client, device, and server and require no requests. 

ElephantSQL (success story – 5) :

ElephantSQL (a PostgreSQL database as a service) is using queues in the architecture for daily backup handling. Once a day there are thousands of “perform-database-backup” messages added to the queue, one for each database. Another service is then subscribing to the messages and creating the backups.

6. Conclusion :

Just because you know how to implement MQ, that won’t make you a best MQ Player, however when and where implementing the MQ in such a way that serves the best fit purpose, will definitely make you one. Happy learning! And always remember in the IT industry change is the only constant thing. So don’t be afraid to replace your current asynchronous task executors with MQ.