Monitor Memory Utilization and Disk Utilization on EC2 instances with Cloudwatch Custom metrics

AWS CloudWatch provides most of the monitoring Metrics by default. But it doesn’t have any metrics for memory utilization details and Disk space uses. So if you want to monitor the memory on your system or monitor free disk space using CloudWatch. Then first you need to add these metrics to your account using custom scripts.

The Amazon CloudWatch Monitoring Scripts for Amazon Elastic Compute Cloud (Amazon EC2) Linux-based instances demonstrate how to produce and consume Amazon CloudWatch custom metrics. These sample Perl scripts comprise a fully functional example that reports memory, swap, and disk space utilization metrics for a Linux instance.dd8255256e8532bbd738a98c1050dd7f

In this post we’ll look at how we can use CloudWatch to monitor these extended metrics, allowing you to build reports and even send alerts to Slack based on set thresholds and defined metric upper bounds.

Supported Systems

These monitoring scripts are intended for use with Amazon EC2 instances running Linux. The scripts have been tested on instances using the following Amazon Machine Images (AMIs), both 32-bit and 64-bit versions:

  • Amazon Linux 2
  • Amazon Linux AMI 2014.09.2 and later
  • Red Hat Enterprise Linux 7.4 and 6.9
  • SUSE Linux Enterprise Server 12
  • Ubuntu Server 16.04 and 14.04

Package Contents

The package for the monitoring scripts contains the following files:

  • CloudWatchClient.pm – Shared Perl module that simplifies calling Amazon CloudWatch from other scripts.
  • mon-put-instance-data.pl – Collects system metrics on an Amazon EC2 instance (memory, swap, disk space utilization) and sends them to Amazon CloudWatch.
  • mon-get-instance-stats.pl – Queries Amazon CloudWatch and displays the most recent utilization statistics for the EC2 instance on which this script is executed.
  • awscreds.template – File template for AWS credentials that stores your access key ID and secret access key.
  • LICENSE.txt – Text file containing the Apache 2.0 license.
  • NOTICE.txt – Copyright notice.

Installing prerequisites on the instance

With some versions of Linux, you must install additional modules before the monitoring scripts will work.

At a command prompt, install packages as follows:

Amazon Linux 2 and Amazon Linux AMI

$ sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64
$ sudo yum install zip unzip

Red Hat Enterprise Linux

$ sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA --enablerepo="rhui-REGION-rhel-server-optional" -y 
$ sudo yum install zip unzip

SUSE Linux Enterprise Server

$ sudo zypper install perl-Switch perl-DateTime
$ sudo zypper install –y "perl(LWP::Protocol::https)"

Debian Based Systems 

$ sudo apt-get update
$ sudo apt-get install unzip libwww-perl libdatetime-perl

Setup an IAM Role to the EC2 Instance 

Associate an IAM role (instance profile) with your instance, verify that it grants permissions to perform the following operations:

  • cloudwatch:PutMetricData
  • cloudwatch:GetMetricStatistics
  • cloudwatch:ListMetrics
  • ec2:DescribeTags

IAMRole1.jpg

ec2role2

ec2role3

ec2role4.JPG

ec2role5.jpg

ec2role6

ec2role7

Policy :

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ec2:DescribeTags",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"
],
"Resource": "*"
}
]
}

Attach role to Ec2 Instance 

ec2role8

You can aldo use the policy below as an inline policy on a user with programmatic access to the cloud through the form of an Secret Key / Access Key.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1491836185126",
            "Action": [
                "cloudwatch:PutMetricData"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "Stmt1491836200583",
            "Action": [
                "cloudwatch:GetMetricStatistics"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "Stmt1491836212902",
            "Action": [
                "cloudwatch:ListMetrics"
            ],
            "Effect": "Allow",
            "Resource": "*"
        },
        {
            "Sid": "Stmt1491836285195",
            "Action": [
                "ec2:DescribeTags"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Installing the monitoring scripts on the instance

At a command prompt, move to a folder where you want to store the monitoring scripts and run the following command to download them and install :

mkdir -p /scripts/cloudwatch/ 
cd /scripts/cloudwatch/ 
curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O
unzip CloudWatchMonitoringScripts-1.2.2.zip && \
rm CloudWatchMonitoringScripts-1.2.2.zip && \
cd aws-scripts-mon

install_script

To collect all available memory metrics and send them to CloudWatch, counting cache and buffer memory as used

./mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-util --mem-used --mem-avail

To set a cron schedule for metrics reported to CloudWatch :

Add the following command to report memory and disk space utilization to CloudWatch every five minutes:

$ sudo crontab -e 

*/5 * * * * /scripts/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --mem-util --disk-space-util --disk-path=/ --from-cron

To get utilization statistics for the last 12 hours, run the following command:

/mon-get-instance-stats.pl --recent-hours=12

Instance metric statistics for the last 12 hours.

CPU Utilization
    Average: 1.06%, Minimum: 0.00%, Maximum: 15.22%

Memory Utilization
    Average: 6.84%, Minimum: 6.82%, Maximum: 6.89%

Swap Utilization
    Average: N/A, Minimum: N/A, Maximum: N/A

Disk Space Utilization on /dev/xvda1 mounted as /
    Average: 9.69%, Minimum: 9.69%, Maximum: 9.69%

Monitor Disk Space and send notification using sns topic 

Create a disk usage script that monitor the disk space and send notification via sns topic

$ mkdir -p /scripts/disk-monitor/ 

$ vi disk-usage.sh

#!/bin/bash

Private_IP=`curl http://169.254.169.254/latest/meta-data/local-ipv4`

Instance_ID=`curl http://169.254.169.254/latest/meta-data/instance-id`

df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;

do

usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )

partition=$(echo $output | awk '{ print $2 }' )

if [ $usep -ge 80 ]

then

aws sns publish --region us-west-2 --topic-arn arn:aws:sns:us-east-1:784334838676:emailnotification --message "$partition is running out of space -  $usep% full PRIVATE_IP=$Private_IP, Instance_ID=$Instance_ID" --subject "PRODUCTION Disk Space Alert - $Private_IP - $partition - $usep%"

fi

done

For –topic-arn provide your own sns topic details

chmod +x /scripts/disk-monitor/disk-usage.sh

Run once to test the script

sh /scripts/disk-monitor/disk-usage.sh

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 12 100 12 0 0 12 0 0:00:01 --:--:-- 0:00:01 3000

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed

100 19 100 19 0 0 19 0 0:00:01 --:--:-- 0:00:01 6333

Set the cron scheduled for disk usage

crontab -e

*/15 * * * * sh /scripts/disk-monitor/disk-usage.sh

 

Creating Dashboard and Set Alarm 

Go to Cloudwatch in  AWS management console and create a dashboard

dashboard1

 

dasboard2

Search the metrics with ec2 instance id here :  i-00ce14375393c4b5b

Select the Linux System Metrics for memory utilization metrics

dasboard3

Configure the widget as below :

dashboard4

For configuring disk utilization select Linuxsystem > File system metrics

dasboard5

Dashboard looks like this :

dashboard6

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Powered by WordPress.com.

Up ↑

%d bloggers like this: