Custom Monitoring System using AWS and ElasticSearch
Monitoring users' activities, system failures, and getting insights is an essential component of any software system.
In this article, I will explain how you can implement a monitoring system using AWS cloud, ElasticSearch, and Kibana, which will provide you with UIs that contain graphs and statistics about your system.
Additionally, emails will be sent when your backend fails to process a request.
Implementing the monitoring system using the following diagram will enable you to extend your system in any way you wish, where you can always add new components easily for various additional functionalities like SMS sending, data analysis, advanced paid monitoring systems, etc…
1- Server-Side
The code examples are written using .Net Core C# which you can replace with your programming language.
When a user hits an API endpoint or browses through your website you are usually interested in basic information that tells you.
1- What functionality was used.
2- Who used it.
3- When.
4- How did it go (Success/Failure)
5- In case of failure, what went wrong.
To construct the above model and send data to the next component of the system (AWS SNS) a middleware can be injected into your pipeline.
Example:
That is all the coding you need to do on the server’s side.
2- Cloud-Side
A- AWS SNS:
SNS serves the purpose of fanning out your system usage data to the various system components interested in such data, it also provides the dynamicity of adding new consumers that process data in different ways to enable various functionalities you need.
In my implementation, I have linked the SNS topic to two other AWS services which are explained next.
B- AWS SES:
Using AWS SES you can send emails to your support team members when a server’s failure happens.
Usually, support teams are only concerned about server errors, so your monitoring system shouldn’t email the support about successfully processed requests.
This can be achieved using the following subscription filter policy, which can be added when creating an SES subscription on SNS side.
{
"State": [
"Failure"
]
}The filter policy verifies that the SNS notification holds a message attribute called “State” with the value “Failure” before forwarding the notification to SES where the email is sent.
Note: message attributes are added using the following code line in middleware.
MessageAttributes = new Dictionary<string, MessageAttributeValue>() { { "State", new MessageAttributeValue() { StringValue = info.State, DataType = "String" } } }C- AWS SQS
To feed another copy of the monitoring data into ElasticSearch you need to create an SQS subscription which is a messaging service that a Lambda function can consume.
D- AWS Lambda
There are several ways that you can ingest data into ElasticSearch from an SQS queue like FunctionBeat, Logstash, or a custom Lambda which is what I did.
You can see the full code of the custom lambda in my git repo here.
Please note that methods proposed by ElasticSearch (FunctionBeat, Logstash) are generally preferred over building your own implementation.
E- ElasticSearch And Kibana
System usage data should be stored in a datastore so teams interested can see the data and generate graphs/statistics of the system usage, failure rates, status codes, etc…
In order to do so, ElasticSeach with Kibana can be used which comes with great built-in capabilities where you can build your customized dashboard that contains the graphs and statistics you need.
You can use AWS ElasticSearch or Elastic Cloud to get a hosted ElasticSearch solution that comes with Kibana by default.
Now that your data is fed into ElasticSearch, you can configure Kibana to build the monitoring dashboard you wish.
Links:
