Getting EC2 Sizing Right with Comprehensive Performance Metrics

May 20 2013 | by Vittaly Tavor

0 Flares Twitter 0 Facebook 0 LinkedIn 0 Email 0 Filament.io 0 Flares ×

To ensure that your EC2 instances are properly sized in accordance with your real needs, clear and comprehensive insight into your EC2 performance metrics is a must.

CloudWatch is a great Amazon service providing basic performance metrics on running instances. The metrics are collected on the hypervisor level, so the existing instance metrics include:

  • CPU utilization
  • I/O (bytes and operations)
  • Network (bytes and operations)

However the memory and swap utilization metrics are missing – these metrics are operating system specific and the hypervisor (at least at Amazon) does not have any visibility into the OS. Of course, without metrics on memory, it’s not so easy to make sizing decisions.

Custom CloudWatch Metrics

Amazon partially overcomes this deficiency by allowing customers to push any custom metrics into CloudWatch. They even wrote a set of very convenient sample scripts to push memory, swap and some other metrics.

Amazon’s original scripts may be found here for Linux and here for Windows
These scripts perform the main job of the data collection on the host, and then pushing these metrics into CloudWatch.

However, Amazon scripts are just a sample: you must manually install a (sometimes long) list of prerequisites. You must manually create a sampling schedule and you must manually do that for any instance you wish to activate the monitoring on.

Making things better with Cloudyn scripts

Cloudyn went one step further. We used Amazon scripts as a base, and wrapped it with a simple one-line installer. Cloudyn scripts check and install the prerequisites on most of the popular platforms, copy Amazon package and add an execution schedule.

In order to allow the automated installation, we made the installers command-line based. The installation needs run as root on Linux, or as an administrative user on Windows.

Cloudyn one-line installation command may be added into your Chef recipe or any other automated configuration script, so that any new machine is launched with the memory monitoring.

To be able to push the CloudWatch metrics you need to make sure to have prepared the credentials for the IAM user with the appropriate permissions. See Appendix A below for the security policy required for this IAM user.

Cloudyn Scripts on Linux

Let’s take a look at how to get things going on Linux.

Installing the package
To install the package on a Linux server, run the one-liner below (substitute the AAA and SSS with the actual IAM user access and secret key) as root:

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | AWSAccessKeyId=AAA AWSSecretKey=SSS Action=Install InstallLWP=Yes sh

This command will install all the prerequisites, download Amazon package to push the custom metrics, configure it and define a schedule to push the metrics every 15 minutes. Amazon script itself won’t be executing as root, but as a user with very limited permissions.
cw-linux-installation-screenshot
Uninstalling the package
To uninstall the package and remove the schedule –

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=Uninstall sh

Checking the metrics for the last 2 hours
To see what metrics were pushed to CloudWatch in the last 2 hours, run –

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=GetCWData sh

2 Pushing-metrics
Getting help/viewing all options
Finally, to see all the usage options, run:

curl http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.sh | Action=Usage sh

Cloudyn Scripts on Windows

Windows installation requires PowerShell 2.0 or higher to run and supports Windows Server 2008 and 2012.

Installing the package
Depending on your command processor, run the appropriate one-liner below (substitute the AAA and SSS with the actual IAM user access and secret key) as an administrative user.

For cmd.exe:

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\"http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\", \"%temp%\1.vbs\"); if($?) {cscript %temp%\1.vbs /I /A AAA /S SSS}

For powershell:

$o=new-object system.net.webclient; $o.downloadfile("http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs", "$env:temp\\1.vbs"); if($?) {cscript "$env:temp\\1.vbs" /I /A AAA /S SSS}

This command will install all the prerequisites, download Amazon package to push the custom metrics, configure it and define a schedule to push the metrics every 15 minutes.
3 installing-the-package-for-Windows

Uninstalling the package
To uninstall the package and remove the schedule –

For cmd.exe:

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\"http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\", \"%temp%\1.vbs\"); if($?) {cscript %temp%\1.vbs /U}

For powershell:

$o=new-object system.net.webclient; $o.downloadfile("http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs", "$env:temp\\1.vbs"); if($?) {cscript "$env:temp\\1.vbs" /U}

Checking the metrics for the last 2 hours
To see what metrics were pushed to CloudWatch in the last 2 hours, run –

For cmd.exe:

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\"http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\", \"%temp%\1.vbs\"); if($?) {cscript %temp%\1.vbs /G}

For powershell:

$o=new-object system.net.webclient; $o.downloadfile("http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs", "$env:temp\\1.vbs"); if($?) {cscript "$env:temp\\1.vbs" /G}

4 checking-the-metrics-for-the-last-2-hours

Getting help/viewing all options
Finally, to see all the usage options, run:

For cmd.exe:

powershell -c $o=new-object system.net.webclient; $o.downloadfile(\"http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs\", \"%temp%\1.vbs\"); if($?) {cscript %temp%\1.vbs /H}

For powershell:

$o=new-object system.net.webclient; $o.downloadfile("http://tools.cloudyn.com/AWS/CW/InstallAWSCWTools.vbs", "$env:temp\\1.vbs"); if($?) {cscript "$env:temp\\1.vbs" /H}

Viewing Memory in Cloudyn Reports

If you elect to utilize CloudWatch for memory metrics, Cloudyn will show these metrics in its reports and will use them in the sizing recommendations. For example, you may see below our memory report:
5 Cloudyn-Memory-Report

In this report it’s clearly visible that the specified instance is using most of its RAM, and occasionally starts using some swap.

How much will it cost me?

At the time of writing (April 2013), Amazon pricing is $0.5 per metric per instance per month. We collect a total of 4 metrics per instance:

– Used Memory (MB)
– Used Memory (%)
– Used Swap (MB)
– Used Swap (%)

This equals $2/month/instance.
(AWS gives you the first 10 custom metrics for free. So for example, if you are collecting these 4 metrics for 10 instances, you only pay for 7.5 instances or $15/month).

By identifying which instance types are your largest cost drivers, you can push custom scripts for just those instances for 1 month (recommended). At that point you should be ready to confidently make sizing decisions based on our recommendations, and uninstall the custom metrics with no further charges by Amazon.

Conclusion

Memory is a necessary metric for any serious instance sizing. Cloudyn brings it all together in a very convenient way, presenting all the metrics together and helping with the sizing decisions.
We recommend periodically defining the memory metrics for short periods of time, analyzing the results and remove the memory metrics to save the cost. Now, with Cloudyn auto-installation scripts, this task is easy.

Appendix A

IAM user credentials required to read/write the CloudWatch metrics:

{
  "Statement": [
    {
      "Sid": "Stmt1359476415731",
      "Action": [
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "cloudwatch:PutMetricData"
      ],
      "Effect": "Allow",
      "Resource": [
        "*"
      ]
    }
  ]
}

 

Discover how Cloudyn can help you optimize your cloud – Sign-Up for FREE Now!

Login

Forgot Password?

No account yet? Register.

0 Flares Twitter 0 Facebook 0 LinkedIn 0 Email 0 Filament.io 0 Flares ×