Tuesday, September 07, 2010
  Search

Contact Us!

AppTheory wants to talk to you about your business requirements.

Click Contact to fill out an online requirements questionnaire.

Thanks!

DNN Gold Partner


 

Ecommerce Award

Subscribe to this Blog

DotNetNuke Blogs

Current Articles | Archives | Search |

Scaling Out DotNetNuke Professional on Amazon EC2

Background

Amazon Web Services (AWS) is a set of cloud based infrastructure web services that are offered to developers and companies.  These services provide expandable and flexible infrastructure to user.  The are several parts to the cloud infrastructure. Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides compute capacity in the cloud. This compute capacity is provided by Amazon Machine Instances (AMIs).  The machine instances can provide cloud based virtual servers with windows and *nix based operating systems.

Amazon EC2 also provides several other features such as Amazon CloudWatch (Amazon CW) a system that monitors the health of your launched AMIs. Elastic Load Balancing (ELB) a virtual load balancer that routes traffic between multiple instances. Auto Scaling (AS) a service that uses Amazon CW’s metrics to scale your AMIs depending on traffic or load on your application.

Goals

The purpose of this post is to deploy and test an auto scaling group on Amazon EC2. The group will consist of a web server that has DotNetNuke Professional installed. There will be a SQL server that will not scale during the test.  There will also be a test server that will place a load on the web server. The auto scaling group should scale out if the CPU utilization for all servers in the auto scaling group are over 60%. The auto scaling group should scale down if the CPU utilization for all servers in the auto scaling are under 30%.  The minimum size of the auto scaling group will be 1 instance and the maximum size will be 5 instances.

This test is purely experimental. The thresholds that have been picked were arbitrarily chosen for a production environment that used AS you would need to carefully test and plan your thresholds. I am not trying to setup a proper DNN web farm.  This test does not take into account file caching and syncing between different web servers.  This test shows the steps to setup a elastic load balancer that auto scales on Amazon EC2. To use this in a DNN environment there needs to be further testing and review.

The following diagram show the basic setup for this test on Amazon.

image

Tools – The following downloads are need to setup and perform the tests.

Downloads Used

Online Tools

  • AWS Management Console – A graphical user interface to manage Amazon EC2, Amazon Elastic Map Reduce, and Amazon CloudFront.

EC2 Setup

The majority of these step can be performed with Amazon EC2 API Tools, Elasticfox, or the AWS Management Console.

    1. Install the above tools.
    2. The first step is to create the security groups for Amazon EC2.

        WEB Group - port 80 open to all IPs
        SQL Group -port 1433 open to the Web Group
        TEST Group - port 80 open to all IPs

    3. The next step is to create the instances needed for the test and assign to the correct security group.

        1 web instance – 32 bit Windows 2003 sp2 with IIS and .NET installed
        1 SQL instance – 64 bit Windows 2003 sp2 with SQL Server 2005 sp2 installed
        1 test instance – 32 bit Windows 2003 sp2

    4. Configure EC2 settings for each instance. Turn off sysprep and new name.
    5. Assign an elastic IP address to your SQL instance. Then Create a host record on the web instance that points to the SQL instance.
    6. Install and setup DNN professional 5.1 on the web instance and SQL instance DNN Professional.
    7. Test that you can reach your DNN installation remotely. In this case I added the portal alias demo.devtheory.com. I then assigned a CNAME that used the public DNS provided by Amazon EC2 for the web instance.
    8. Bundle and register an AMI for the web instance and the SQL instance.
    9. Install Pylot on test instance Pylot install guide.
    10. Bundle and register an AMI for test instance.

Elastic Load Balancing and Auto Scaling Setup

These steps require the use of the Elastic Load Balancing API Tools and the Auto Scaling API Tools. These are command line tools that can be run on linux or windows.  When I installed them there was a problem with the tools passing the path correctly. To fix this I went into the bin of both tool sets and altered one command. For the Elastic Load Balancing API tools alter elb-cmd.cmd  and for the Auto Scaling API Tools alter as-cmd.cmd. The line you are looking for is:

“%JAVA_HOME%\bin\java%" %SERVICE_JVM_ARGS% -classPath %CP% com.amazon.webservice.cli %CMD% %ARGV%

I added the quotes to make the path work correctly. I would also recommend testing each of the commands and creating a batch that you can alter for adding a ELB and AS groups.  The commands can be lengthy and easy to mistype.

      elb-create-lb  prolb --headers --listener "lb-port=80,instance-port=80,protocol=http" --availability-zones us-east-1d
      as-create-launch-config prodLaunch --image-id ami-fc54b795 --instance-type m1.small --group WEB
      as-create-auto-scaling-group proASG --launch-configuration prodLaunch --availability-zones us-east-1d  --min-size 1 --max-size 5 --load-balancers prolb
      as-create-or-update-trigger Trigger --auto-scaling-group proASG --namespace "AWS/EC2" --dimensions "AutoScalingGroupName=proASG" --measure CPUUtilization --statistic Average --period 60 --breach-duration 120 --lower-threshold 30 --lower-breach-increment"=-1" --upper-threshold 60 --upper-breach-increment 1

The above commands create:

    1. A load balancer named “prolb”
    2. An auto scaling launch configuration called “prodLaunch” which creates new m1.small instances from ami-fc54b795 and assigns them to WEB group.
    3. An auto scaling group called “proASG” the group is located in us-east-1d availability zone, a minimum size of 1 instance, a maximum size of 5 instances, and uses the “prolb” load balancer.
    4. This will also launch a web instance that will become part of the Auto scaling group since the minimum group size is 1.
    5. A trigger named “Trigger” that looks at the “proASG” auto scaling group will check the Average CPU load for the all instances in the auto scaling group every minute. If the CPU load is greater than 60% for two minutes it will launch an instance that is defined by “prodLaunch.” If the CPU load is less than 30% for two minutes it will terminate an instance. 

Testing Elastic Load Balancing and Auto Scaling

This is the basic test that was run to test AS on Amazon EC2

    1. Set DNS CNAME for your site to the load balancer URL. Give time for propagation.
    2. Get a baseline of the load that is being reported on the Auto Scaling Group with the CloudWatch API tools the following command tell the percentage CPU load for the last hour taken every minute. 
      C:\Windows\system32>mon-get-stats CPUUtilization --period 60 --statistics "Avera
      ge"       --namespace "AWS/EC2" --dimensions "AutoScalingGroupName=proASG"
      2009-11-09 18:22:00  1.0  2.19  Percent
      2009-11-09 18:23:00  1.0  3.07  Percent
      2009-11-09 18:24:00  1.0  2.19  Percent
      2009-11-09 18:25:00  1.0  2.63  Percent
      2009-11-09 18:26:00  1.0  2.19  Percent
      2009-11-09 18:27:00  1.0  2.63  Percent
      2009-11-09 18:28:00  1.0  2.19  Percent
      2009-11-09 18:29:00  1.0  2.59  Percent
      2009-11-09 18:30:00  1.0  2.23  Percent
      2009-11-09 18:31:00  1.0  2.63  Percent
      2009-11-09 18:32:00  1.0  2.19  Percent
      2009-11-09 18:33:00  1.0  2.63  Percent
      2009-11-09 18:34:00  1.0  2.16  Percent
      2009-11-09 18:35:00  1.0  2.68  Percent
      2009-11-09 18:36:00  1.0  2.19  Percent
      2009-11-09 18:37:00  1.0  2.63  Percent
      2009-11-09 18:38:00  1.0  2.19  Percent
      2009-11-09 18:39:00  1.0  2.63  Percent
      2009-11-09 18:40:00  1.0  2.19  Percent
      2009-11-09 18:41:00  1.0  2.59  Percent
      2009-11-09 18:42:00  1.0  2.23  Percent
      2009-11-09 18:43:00  1.0  2.19  Percent
      2009-11-09 18:44:00  1.0  2.59  Percent
      2009-11-09 18:45:00  1.0  2.23  Percent
      2009-11-09 18:46:00  1.0  2.63  Percent
      2009-11-09 18:47:00  1.0  2.19  Percent
      2009-11-09 18:48:00  1.0  2.19  Percent
      2009-11-09 18:49:00  1.0  2.63  Percent
      2009-11-09 18:50:00  1.0  2.19  Percent
      2009-11-09 18:51:00  1.0  2.19  Percent
      2009-11-09 18:52:00  1.0  8.33  Percent
      2009-11-09 18:53:00  1.0  2.63  Percent
      2009-11-09 18:54:00  1.0  2.63  Percent
      2009-11-09 18:55:00  1.0  2.19  Percent
      2009-11-09 18:56:00  1.0  2.59  Percent
      2009-11-09 18:57:00  1.0  2.23  Percent
      2009-11-09 18:58:00  1.0  3.02  Percent
      2009-11-09 18:59:00  1.0  2.23  Percent
      2009-11-09 19:00:00  1.0  2.63  Percent
      2009-11-09 19:01:00  1.0  2.63  Percent
      2009-11-09 19:02:00  1.0  2.63  Percent
      2009-11-09 19:03:00  1.0  2.63  Percent
      2009-11-09 19:04:00  1.0  2.63  Percent
      2009-11-09 19:05:00  1.0  2.19  Percent
      2009-11-09 19:06:00  1.0  2.63  Percent
      2009-11-09 19:07:00  1.0  2.63  Percent
      2009-11-09 19:08:00  1.0  2.59  Percent
      2009-11-09 19:09:00  1.0  2.23  Percent
      2009-11-09 19:10:00  1.0  2.63  Percent
      2009-11-09 19:11:00  1.0  2.63  Percent
      2009-11-09 19:12:00  1.0  2.19  Percent
      2009-11-09 19:13:00  1.0  3.07  Percent
      2009-11-09 19:14:00  1.0  2.63  Percent
      2009-11-09 19:15:00  1.0  2.19  Percent
      2009-11-09 19:16:00  1.0  2.16  Percent
      2009-11-09 19:17:00  1.0  2.23  Percent
      2009-11-09 19:18:00  1.0  2.63  Percent
      2009-11-09 19:19:00  1.0  2.19  Percent
      2009-11-09 19:20:00  1.0  2.63  Percent

    3. Query the number of instances you have open at start – there are 3 instances running before the start 
      C:\Windows\system32>ec2-describe-instances
      RESERVATION     r-55a9683d      400000000009    AWSDEV.TEST
      INSTANCE        i-53e6773b      ami-dd20c3b4    ec2-174-000-168-000.compute-1.am
      azonaws.com     ip-10-000-87-000.ec2.internal   running Dev key 0
      m1.large        2009-11-05T19:59:02+0000        us-east-1d
      windows monitoring-disabled     174.000.168.000 10.000.87.000
      RESERVATION     r-91c607f9      400000000009    AWSDEV.SQL
      INSTANCE        i-b38a1bdb      ami-49628120    ec2-174-000-32-000.compute-1.amaz
      onaws.com       ip-10-000-230-000.ec2.internal   running Dev key 0
      m1.large        2009-11-05T21:38:51+0000        us-east-1d
      windows monitoring-enabled      174.000.32.000   10.000.230.000
      RESERVATION     r-310cc159      400000000009    WEB
      INSTANCE        i-a956cac1      ami-fc54b795    ec2-75-000-252-000.compute-1.ama
      zonaws.com      ip-10-000-77-000.ec2.internal   running         0
      m1.small        2009-11-09T07:31:38+0000        us-east-1d
      windows monitoring-enabled      75.000.252.000  10.000.77.000

    4. Start the load testing for your site URL – http://demo.devtheory.com. I used 100 agents, no ramp up, Interval 250 ms, and duration 1200 seconds.
    5. Use either the Management Console or commands from step 1 and 2 to view current metrics and any instances that have been launched or terminated.

Test Results

During several tests with a duration of 20 minutes here were the general results.

    1. Original web instance will show CPU utilization over 60%.
    2. proASG Auto Scaling Group will scale out two more web servers.
    3. The scale out will not complete during the test window.
    4. Scaling down will not occur until the after the last web server has reported that it is started and ready.
    5. When scaling down the oldest web instances are the ones to be terminated first.
    6. I noticed on a few test runs that when trying to reach the web site during a scale down I would get a ‘not found’ error. This could be from the load balancer sending the request to a web instance that has just terminated.

Conclusions

The basics seem to work with Auto Scaling and with Elastic Load Balancing.  Amazon CloudWatch seems to give some limited monitoring of instances that can be used to auto scale. The next step would be to see what needs to been done to setup a DNN professional farm on Amazon EC2. The beauty of Amazon Web Services and Amazon EC2 is the ability to stop looking at your application infrastructure as a classic datacenter or hosted application. With Amazon you can easily scale out and to stop being concerned about fixing hardware issues. If the hardware breaks just spin up another instance to replace the failing instance. There are some concerns trying to setup a DNN professional farm that will take advantage of Amazon EC2:

    1. SSL termination – use of HAProxy in place of the Amazon ELB 
    2. handling of session state is it managed by the load balancer
    3. synching and mirroring web files and folders or using a shared elastic block storage
    4. more metrics for CloudWatch - possibly the ability to  create your own at least the ability to check versus a specific time
    5. Better launch time of instances in an auto scale group - two instances in 20 minutes seems a bit slow
    6. To easy to cause a site that is using auto scaling to automatically scale out.  This could get costly. There needs to be a way to detect, determine, and stop an IP that is making too many requests on your site.

I will begin exploring some of these options and see about Amazon EC2s ability to host a DNN professional farm.


Posted in: Cloud Computing on Monday, November 09, 2009 9:21 PM by Max Schneider
Actions: E-mail | Permalink | Comments (0) RSS comment feed

COMMENTS

Only registered users may post comments.
Copyright © 2010 AppTheory