Hi, if you are developing your product using Microservice Architecture, then, you probably familiar with “Event-driven Architecture” as well. But, do you implement it in a good way or in a bad way?
let’s go back a little bit into history. So, this is an old-school diagram that we use to display data in real-time. So, suppose we have a messaging service, message here means like your facebook account messenger or maybe your Instagram DM. The goal is “How do we show the latest data from Back-end to messaging service”, so in this case, it will be front-end. The old way is we have some kind of “API” request to do the back-end every 15 seconds or whatever the configuration is. In this way, we will always call the API and from that API we always output the latest data into the message service. So, that’s what we do for the web application or maybe let’s say the mobile app or something. So, with the same approach, let’s move into “how do we implement this into the Microservice?”.
Let’s say you have analytic service, where, when users dissabihavior in your website, the event and everything, will be start inside this specific service and you have another service called the Machine learning Model. Machine learning Model every 3 hours, will do some API request to the analytic to grab the latest data from the analytic service. So, that’s kind of the same approach, but this specific approach is within the Microservice level.
From this implementation you can see that it’s really not an efficient way to do that. For just displaying the real-time data that should be fine, because you can grab whatever the latest data displayed inside this service, but for the Microservice one, that will be little bit complicated because with the ML Model do the request to analytic every 3 hours, they have to slice what the latest data they get from the previous API-call, how to manage the stage and everything, so that will be a little bit more complicated inside the Microservice level.
So, the new way to do this is by using “Event-Driven Architecture” instead of the message app, do an API request to back-end every 15 seconds, let’s try to do the other way, whenever back-end has new data of the message, it will push the data into the message app. So, message app here will be a passive service just to receive the new data and displayed it into the product or maybe service.
This is the kind of the same approach what your phone notification work, this is what we call the push notification. If you pull the same scenario into the Microservice level, pretty much the same approach, so when analytic service has new analytic data, it will push the data into the Machine Learning Model here. So, now Machine Learning model only has to care about the new data with how, basically what it datas checking, what is the latest data from API-call, I mean the previous API-call, and right now basically, there’s no worries at all for that, just store the new data and process it into the Machine Learning Model.
So, the push method, could be done in multiple ways. Some basic and common implementation is VIA internal API-call between services or maybe Remote Procedure Call between those services. But the problem, one of the biggest problem with this implementation is when you try to sent the data from the back-end to message app and at the time, the message app is down, once the event sent into the message app it would be failed, because basically the service is done and can’t receive any data.
But, “How is the way that you can maintain or handle this specific problem?” You could probably, if this failed, you could probably just retry to send the data again. That’s fine if you do everything in asynchronous way, but if you do everything in synchronous way that will be a big problem because user has to wait until this specific push to the succeed, then they continue to the next process.
Another problem occur by using this specific implementation is when, let’s say you have new service that require to receive the message data from back-end, so that will be another push. With API or Remote Procedure Call, you have to change code inside the back-end from previous one that you only need to send to one specific service, then you have to change the code, now you have 2 lines of code, first one who send to message app service and second one to send to the new service. So, this is quite hard to maintain as well.
Our favorite way to do the “Event-Driven Architecture” is this way, on this specific diagram. So, between 2 services, analytic and Machine Learning Model, we will have another service in between them. This is what we called the message broker.
Normally we will use Kafka, it will say, kind of a lot of big data to send or for last data we will more preferred to use RabbitMQ for the Message Broker. this is the updated diagram when we start using message broker. The analytic service will send the data into message broker first and message broker will send the data into related service that need this data, in this specific case is Machine learning Model.
Okay, let’s really understand how everything works together. The first thing that you have to know is What is the producer and What is consumer.
- Producer basically a service that produce the data and send it into message broker. In our case here, the producer is the analytic service.
- consumer is the service that receive the data from message broker. In our case here is the Machine learning Model, so this is what we call the consumer.
The next thing that you have to understand is about topic and subscribe, so let’s go with topic.
If you try to imagine an email, each email has a subject, and you can assume the topic here is the subject of that email. every data that sent from producer, must be under a specific topic. let’s say producer send a click data, under a specific topic, let’s say click event and producer only care about sending the data into the message broker. The rest is what message broker will handle.
Since producer doesn’t care about where the message or the data will be send to, it’s now that consumer has to take care about that. In order to receive specific data from producer, consumer has to subscribe to specific topic, here. So, this will solve the problem when you have new consumer with API Integration or internal API Integration that we talked before, they have to update the producer stuff. But, with message broker when you have new service that need the same data, you can just subscribe to the same topic that required. So, it will be automatically message broker will also send the data into the new service.
now it’s more easy to maintain your whole infrastructure when new service need the same data from the producer. You don’t have to go to your producer code and update your code to send the same data to new service. With message broker when new service subscribe to specific topic, then that will be automatically receive the data out of the box.
Problem When The Service Is Down
What about the other problem when the service is down, how do we maintain it?
Let’s say the producer send a specific data, to message broker and we have 2 subscribers here, it will send the first data and for both, since both are still up. and here comes the second data, since both are still up as well, it will receive the second data. and suddenly our new service here, stop working. When the time it stop working, message broker will imidiatelly know “ohh, this specific service is down now.” and producer want to send another data which is the third data, since it’s up, it will receive the third data, but since it down, it won’t receive anything, because it’s basically down. and producer also want to send another data, the fourth data. At after sometimes, the new service is up and running again. So, what will happen is message broker will check “oohh, now this service is run what are the data has been sent from producer that this specific service hasn’t receive yet.” and by that, message broker will send the data for 3 and 4. So, that’s how it solved the problem when your service is down and message broker still very smart to handle that specific problem.
Alright, that’s how we implement Event-Driven Architecture inside our project. If you have any thoughts about how you run yours, you can put it inside the comment. To get more popular information, check others article in here.