How to use CloudWatch to generate alerts from logs

If we have multiple servers running a database (mongo) and  you want to put all the error/access logs on a centralized place so that you can troubleshoot your system in case of any error after getting alert notification configured on your logs .

Here is an amazing feature of Amazon Web Services Cloud Suite by which you can achieve this task.Print

We can generate alarms in near real-time every time specific keywords are found in the logs. This type of alarms are useful when you need to find out the moment something happens. For example, an alarm can be generated when an application throws an exception or a critical transaction fails.

We can also detect conditions where a single log entry may be normal but multiple entries within a short time may indicate a problem. A typical example is failed login attempts. Many consecutive failed login attempts for a user within a short time may indicate a brute force attack to gain access to the system.

We can use filters to extract values from space delimited logs, and create alarms based on these values. For example, if metrics like number of connections or response times are logged by an application, you can extract this data from the logs and generate an alarm if the value is above a threshold, etc.

Using the CloudWatch Logs agent, you can publish log data from Amazon EC2 instances running Linux or Windows Server, and logged events from AWS CloudTrail. We recommend instead using the CloudWatch unified agent to publish your log data.

The CloudWatch Logs agent requires Python version 2.7, 3.0, or 3.3, and any of the following versions of Linux:

  • Amazon Linux version 2014.03.02 or later
  • Ubuntu Server version 12.04, 14.04, or 16.04
  • CentOS version 6, 6.3, 6.4, 6.5, or 7.0
  • Red Hat Enterprise Linux (RHEL) version 6.5 or 7.0
  • Debian 8.0

The CloudWatch Logs agent supports IAM roles and users. If your instance already has an IAM role associated with it, make sure that you include the IAM policy below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*"
    ]
  }
 ]
}

Install and Configure CloudWatch Logs on an Existing Amazon EC2 Instance

Starting with Amazon Linux AMI 2014.09, the CloudWatch Logs agent is available as an RPM installation with the awslogs package. Earlier versions of Amazon Linux can access the awslogs package by updating their instance with the sudo yum update -y command.

sudo yum update -y
sudo yum install -y awslogs

Edit the /etc/awslogs/awslogs.conf file to configure the logs to track.

Add following lines to the config file

[/var/log/syslog]
file = /var/log/mongodb/mongod.log
log_group_name = mongo_server
log_stream_name = mongo_server_logs
datetime_format = %b %d %H:%M:%S
time_zone =       UTC

Parameters in agent file :

1.file :- The file specifies the file in which your actual logs are stored on your EC2 instances. This is the log file whose content you want to push on CloudWatch logs . I want to push my nginx access logs onto the CloudWatch so I am specifiying the path of nginx access log file.

2.log_group_name :- It refers to the destination log group. A log group will be created automatically if no log group exists in your CloudWatch.

3.log_stream_name :- It refers to the destination log stream.A log stream can be {instance_id}, {hostname}, {ip_address} or a combination of these.

4.datetime_format :- It specifies how the timestamp is extracted from logs.

%b specifies month (Jan,Feb..)

%d specifies day of month (01,02..)

%H specifies Hour (24-hour clock)

%M specifies Minutes (01,02..59)

%S specifies Seconds (01,02..59)

By default, the /etc/awslogs/awscli.conf points to the us-east-1 region. To push your logs to a different region, edit the awscli.conf file and specify that region.

Start the awslogs service.

sudo systemctl start awslogsd
sudo systemctl enable awslogsd.service

(Optional) Check the /var/log/awslogs.log file for errors logged when starting the service.

We should see the newly created log group and log stream in the CloudWatch console after the agent has been running for a few moments.

logstream1

To install and configure CloudWatch Logs on an existing Ubuntu Server, CentOS, or Red Hat instance

Run the CloudWatch Logs agent installer using one of two options. You can run it directly from the internet, or download the files and run it standalone.

curl https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py -O
sudo python ./awslogs-agent-setup.py --region us-east-1

If the preceding command does not work, try the following:

sudo python3 ./awslogs-agent-setup.py --region us-east-1

We can also execute the awslogs.conf file entries from s3 bucket using below shell script

#!/bin/bash
wget https://s3.amazonaws.com/aws-cloudwatch/downloads/latest/awslogs-agent-setup.py
chmod +x ./awslogs-agent-setup.py
./awslogs-agent-setup.py -n -r us-east-1 -c s3://your_bucket_name/Cloudwatch_agent_conf
sudo service awslogs restart

Searching and Filtering log data. 

Navigate to Log Groups ( here mongo_server) and select the stream log

loggroup1

Click the stream name and see the log data. Now we’ve successfully sent logs to CloudWatch.

errorlogstream1

Creating Metric Filters 

You can use metric filters to search for and match terms, phrases, or values in your log events. When a metric filter finds one of the terms, phrases, or values in your log events, you can increment the value of a CloudWatch metric. For example, you can create a metric filter to search for and count the occurrence of the word ERROR in your log events.

To define metrics for the log group, return to the “Log Groups”, select mongo_server and click on Create Metric Filter

error_pattern1

error_metric1

You should now see the newly created filter. Click on the “Create Alarm” button.

addmetric1

Create an alert

alert1

alert2

Use the SNS you have defined as the notification target for the action (Send notification to field)

CloudWatch is now configured to generate an alarm

alert3.JPG

Sample Alert Notification

alert4

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Powered by WordPress.com.

Up ↑

%d bloggers like this: