5 Steps to Monitor Lightsail VM using Amazon Cloud Watch

Amazon Cloud Watch, Azure Monitor comes under native cloud monitoring tools. Project managers finds it is easier to subscribe to such tools as some of these monitoring features comes along with their cloud subscription. It is a debate whether native cloud monitoring is sufficient to achieve observability and how far they assist the SRE engineers to achieve their MELT goal. While this is a matter of detailed discussion, we shall keep the scope of this post to monitor AWS Lightsail VMs using Amazon Cloud Watch.

This comes under metrics pillar of MELT framework. If you have not read my introductory post about MELT, you may find it here.

Amazon EC2 VMs are bundled with cloud watch agents. They will get monitored in Cloud watch without much hassle. Lightsail is for different audience. I written about it earlier. You may see Is Amazon Lightsail popular? Who are the target users? later. But it is enough for now to know that Lightsail VMs do not come with AWS CloudWatch agents. So they will not be monitored in CloudWatch by default. This post will be helpful for those who prefer native cloud monitoring for AWS LightSail VMs.

It is a 5 step process.

1. Installation of AWS CLI
2. Configuration of AWS CLI
3. Installation of AWS CloudWatch agent
4. Configuration of CloudWatch agent
5. Monitor in CloudWatch

All right. Let’s get started.

Step 1: Install AWS CLI

I used Ubuntu VM for this tutorial. AWS CLI is a light weight command line interface tool to manage AWS infrastructure. AWS CloudWatch agent uses this CLI to monitor the VM and sends the metrics to CloudWatch. Hence installation of AWS CLI is the first step. Execute the of the following command will do that.

sudo apt-get install awscli

You may get some text rolling in your screen like this. Wait until the updates are completed and CLI is installed.

0 upgraded, 93 newly installed, 0 to remove and 93 not upgraded.
Need to get 46.0 MB of archives.
After this operation, 225 MB of additional disk space will be used.
Do you want to continue? [Y/n]

During this update, I found my VM was running with older kernel version. Hence it prompted me for an upgrade.

Kernel upgrade requires a reboot. So I need to reboot my VM after this upgrade.

$ sudo reboot

To ensure AWS CLI is installed properly, you may execute these basic commands.

$ which aws
/usr/bin/aws
$ aws --version
aws-cli/1.22.34 Python/3.10.12 Linux/6.2.0-1018-aws botocore/1.23.34

Step 2: Configure AWS CLI

Next part is to create an access key and secret key in AWS Identity management. You may follow the sequence to create the same.

Login to your AWS console. Search for IAM module. It is the Identity and Access Management module. AWS CLI needs these keys to communicate back to AWS infrastructure.

After landing in IAM console, create users link in the left menu to create a new user for our AWS CloudWatch agent. Name the user as lightsail-cloudwatch-agent. It does *not* need access to AWS management console. It is discouraged to reuse same user for multiple roles. Hence create distinct users for different responsibilities of the organization.

Next part is to assign suitable permissions for the AWS agent to work. You will find 3 options –

1. Add user to group
2. Copy permissions
3. Attach policies directly

Choose Attach policies directly.

Underneath, you will find the permission policies list. Search for CloudWatchAgentServerPolicy. The policy will be listed as the search result. Select it and proceed to next step.

Next step is to review and create the user. Just proceed to next step.

You will see the new user got created.

Click on the lightsail-cloudwatch-agent to create the keys.

Press create access key link to create an access key for our CloudWatch agent. You may reuse these keys for agents on multiple VMs.

Choose the use case as Command Line Interface (CLI).

You will option to download/copy your access and secret keys in AWS. You may store it safely as exposing such sensitive information will be detrimental.

If you have noted down your secret and access keys, you may proceed to next step.

Step 3: Install AWS CloudWatch agent

Amazon CloudWatch agent is available in different packages such as RPM for RedHat flavors, DEB for Debian flavors. I used Ubuntu for this tutorial which is a minimal version of Debian. If you use CentOS or RHEL, you may choose rpm pack.

Download the agent and install it using the following commands

$ wget https://amazoncloudwatch-agent.s3.amazonaws.com/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
$ sudo dpkg -i -E ./amazon-cloudwatch-agent.deb

Step 4: Configure CloudWatch Agent

We are in important face where we link AWS CloudWatch agent and AWS CLI. Execute the following CLI command to configure a profile for monitoring purpose. Enter the access and secret keys without any error like trailing spaces. Region name and output format are optional. You may just press enter.

$ sudo aws configure --profile AmazonCloudWatchAgent
AWS Access Key ID [None]: abc
AWS Secret Access Key [None]: xyz
Default region name [None]: ap-southeast-1
Default output format [None]:

You need to specify what metrics you want CloudWatch Agent to collect and at what interval. You need to specify this in a json format. I choose to collect mem_used_percent at 60 seconds interval. Create a file /opt/aws/amazon-cloudwatch-agent/bin/config.json to do this.

$ sudo vi /opt/aws/amazon-cloudwatch-agent/bin/config.json

{
"agent": {
  "metrics_collection_interval": 60,
  "run_as_user": "root"
},
"metrics": {
  "append_dimensions": {
  "ImageID": "${aws:ImageId}",
  "InstanceId":"${aws:InstanceId}",
  "InstanceType":"${aws:InstanceType}"
},
"metrics_collected": {
  "mem": {
    "measurement": [
    "mem_used_percent"
    ],
  "metrics_collection_interval": 60
  }
}
}
}

After creating the file, we shall create common-config.toml file

In AWS, common-config.toml is a configuration file used by the CloudWatch agent to set shared settings that apply to both CloudWatch monitoring. It’s typically located in the following paths:

• Linux: /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
• Windows Server: C:\ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Create this file and enter your profile name, which was created above.

$ sudo vi /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
[credentials]
shared_credential_profile = "AmazonCloudWatchAgent"

The amazon-cloudwatch-agent-ctl command is a powerful tool for managing the Amazon CloudWatch Agent. It allows you to perform several key tasks related to the agent’s configuration, operation, and troubleshooting. We invoke this command to specify out configuration and start monitoring the VM.

$ sudo amazon-cloudwatch-agent-ctl -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -a fetch-config -s

You may get an output something like this, based on your configurations.

processing amazon-cloudwatch-agent
Got Home directory: /root I! Set home dir Linux: /root I! SDKRegionWithCredsMap region: ap-southeast-1 Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp
Start configuration validation…
2024/01/31 19:23:39 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_config.json.tmp …
2024/01/31 19:23:39 I! Valid Json input schema.
2024/01/31 19:23:39 D! ec2tagger processor required because append_dimensions is set
2024/01/31 19:23:39 D! pipeline hostDeltaMetrics has no receivers
2024/01/31 19:23:39 Configuration validation first phase succeeded
I! Detecting run_as_user…
Got Home directory: /root
I! Set home dir Linux: /root
I! SDKRegionWithCredsMap region: ap-southeast-1
Got Home directory: /root
D! [EC2] Found active network interface
I! imds retry client will retry 1 times
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded
amazon-cloudwatch-agent has already been stopped
Created symlink /etc/systemd/system/multi-user.target.wants/amazon-cloudwatch-agent.service → /etc/systemd/system/amazon-cloudwatch-agent.service.

If there is are no errors, agent should be started by now. You may verify using the same amazon-cloudwatch-agent-ctl command.

$ sudo amazon-cloudwatch-agent-ctl -a status
{
"status": "running",
"starttime": "2024-01-31T11:23:40+00:00",
"configstatus": "configured",
"version": "1.300032.3b392"
}

Step 5: Monitor in CloudWatch console

Ok, Here is the final step, to start monitoring our virtual machine.

Login to AWS console. Search for CloudWatch. Open CloudWatch window. The new VM may not appear. You need to configure CloudWatch console to get the metrics from custom name spaces. To do that, expand the CloudWatch menu in the left, and go to Metrics > All metrics.

You will see an empty graph. You will see Browse option, where custom namespaces are given. You will see CWAgent is listed. Click it.

Select the metric of our VM, to view the memory utilization graph.

This tracks the memory utilization of the LightSail VM on 60 seconds interval. You may add new dashboard and add this as a widget.

CloudWatch is not entirely free as you think. Look at Amazon pricing and consume as per your available budget.

How to add additional metrics to CloudWatch Agent?

We collected only one metric mem_used_percent in this example. How to collect more metrics. You have a list of metrics supported by AWS CloudWatch agent here. I used two more additional memory metrics in the below configuration. Edit the config file.

sudo vi /opt/aws/amazon-cloudwatch-agent/bin/config.json

{
"agent": {
"metrics_collection_interval": 60,
"run_as_user": "root"
},
"metrics": {
"append_dimensions": {
"ImageID": "${aws:ImageId}",
"InstanceId":"${aws:InstanceId}",
"InstanceType":"${aws:InstanceType}"
},
"metrics_collected": {
"mem": {
"measurement": [
"mem_used_percent",
"mem_total",
"mem_used"
],
"metrics_collection_interval": 60
}
}
}
}

Activate the config

sudo amazon-cloudwatch-agent-ctl -c file:/opt/aws/amazon-cloudwatch-agent/bin/config.json -a fetch-config -s

Make sure that the agent is running. If your JSON is invalid, the agent won’t start. You can use sites like jsonlint to check your JSON format. Otherwise, small mistakes can cost you valuable time.

sudo amazon-cloudwatch-agent-ctl -a status
{
"status": "running",
"starttime": "2024-02-03T17:44:17+00:00",
"configstatus": "configured",
"version": "1.300032.3b392"
}

Check the CloudWatch now, You will see the mem_used, and mem_total appear now.

I asked myself, why memory alone? Why don’t we monitor other metrics also, like netstat, network consumption. Here is the configuration file.

{
“agent”: {
“metrics_collection_interval”: 60,
“run_as_user”: “root”
},
“metrics”: {
“append_dimensions”: {
“ImageID”: “${aws:ImageId}”,
“InstanceId”: “${aws:InstanceId}”,
“InstanceType”: “${aws:InstanceType}”
},
“metrics_collected”: {
“mem“: {
“measurement”: [
“mem_used_percent”,
“mem_total”,
“mem_used”
],
“metrics_collection_interval”: 60
},
“net“: {
“measurement”: [
“net_bytes_recv”,
“net_bytes_sent”
],
“metrics_collection_interval”: 60
},
“netstat“: {
“measurement”: [
“netstat_tcp_established”,
“netstat_tcp_listen”,
“netstat_tcp_time_wait”
],
“metrics_collection_interval”: 60
}
}
}
}

The net metrics do not appear in AWS yet. I need to troubleshoot. I’ll update this post, if I’m able to collect them successfully.

Is there a simple way to monitor that requires less effort? Yes. I’ll write it in another post.

—

This post is written as part of #WriteAPageADay campaign of BlogChatter

Image by Freepik