Optimizing the Stack Template

My working stack, that is able to connect to the instance, has the following design template:

Overview of Design Template -No adjustments

My first focus in refining this stack is to remove any objects that are useless due to not being connected in the system.

This includes two EC2-route tables that aren’t attached within the VPCs, and two of the three elastic IPs, as they were also not necessary.

The updated template is as follows:
Updated Design template

I had originally thought that that the unattached security groups could also be removed from the VPCs, but when I attempted to rebuild the stack, there were dependency errors that forced me to put the security groups back into the JSON script.

My next step was to personalize the stack. This has been done by renaming the stack’s instance. I renamed it to “StackInstance” for easy identification.
EC2 renamed-Stack

My final attempt in optimizing the template, was to organize the stack instance to automatically connect with the elastic IP. This was done by using the following JSON script from: AWS::EC2::EIPAssociation

{
“Type”: “AWS::EC2::EIPAssociation”,
“Properties”: {
“AllocationId”: String,
“EIP”: String,
“InstanceId”: String,
“NetworkInterfaceId”: String,
“PrivateIpAddress”: String
}
}

This however, was unsuccessful as I couldn’t determine what was required for AllocationId, and so I have left it as a manual set-up.

That was every further adjustment done upon the CloudFormer template.

Connecting to the Stack’s Instance

Now that the stack’s instance is at a lower charge rate, I will try to connect to the t2.micro instance.

My first attempt resolved in the following error:

003 Still not able to access _Error

The first thing that I noticed, was that I didn’t have a public IP attached to the instance.
003 CFTemplate EC2 Instance Info

When I looked into the elastic IP section of the EC2 service, there were none there. As these are important in allowing me to connect to the RDGW instance, I added the JSON script for elastic IPs into the CloudFormer Template that I was using. The reason that they weren’t there, was because I had accidentally left them out during the CloudFormer template stack creation process.

When I ran the CloudFormer stack again, the RDGW instance still didn’t have a public IP, however, the elastic IPs had been created, so it was only a matter of manually associating the IP to the instance.

I then tried to connect again, but the connection still failed, with the same error response showing.

I considered then, that it may be an issue with the security groups attached to the stack’s VPC. My initial response was to adjust the JSON script as set all of the security groups’ ingress IPs to 0.0.0.0/0. This action was taken because I wanted to make everything open as a means of determining whether or not the failure to connect was due to the security groups. My next attempt to connect was still unsuccessful, which determined that it was not a security issue. Because this was ruled out, I replaced the security group IPs back to their original addresses for best practice purposes.

My next consideration was to utilize my other CloudFormer template that only had the single VPC in its design. This was to determine whether there may have been a CloudFormer template construction issue that was resulting in the connection failure. This however, was not the case as the single VPC template also failed to connect.

My final attempt, was to change the WiFi connection that I was using. This is because NMIT has to potential networks, both with different firewall settings, and the connection that I use has been known to not allow a remote connection to occur. This also, was unsuccessful, as was my attempt with my home network.

With all of these potential connection errors having been ruled out, I sought help from my classmates as to the design template of their successful stacks, as this would enable me to compare my stack’s design template and see the difference in design that was causing my connection error. While helping me with providing their design template, one of the classmates suggested that I try create a new CloudFormer template from the Microsoft’s Quick Start, Scenario 3, and create the stack to be completely open to any RDP IP addresses.

I did as he suggested, and during the creation of the initial stacks from that are based from scenario 3, and set the Network Configuration for ‘Allowed Remote Desktop Gateway External Access CIDR’ to 0.0.0.0/0.
005 Specify Details NS AZ_Options

Once the stack results were organized through the CloudFormer, I ran the new template, removing any of the errors in the JSON script that were causing the stack to rollback. Once the stack was complete, I attached an elastic IP to the instance, and attempted to connect to it. The result was successful.
01 ADDS RDP Success

The previous failures were due to a discrepancy from the remote desktop gateway external access CIDR that had been set-up with the creation of the stacks prior to CloudFormer. Once that had been resolved, the connection was available.

Running the CloudFormer Template

This blog post follows on from the previous post: Adjusting the CloudFormer Template

After the first stack creation of the CloudFormer template, the rollback errors provide information on what needs to be adjusted in the JSON script.

The following list contains the ‘CREATE_FAIL’ events, and my method in resolving these errors.

Adjustments to the CF Template

  • FAIL: dbsubnetdefaultvpc91a918f5; Some input subnets (subnet-1ab50b43, subnet-93532cf7) are invalid.
    • Attempting to remove from the JSON script:
      “SubnetIds”: [
      “subnet-1ab50b43”,
      “subnet-93532cf7”
      ]
    • The first attempt was unsuccessful, so attempting to delete the entire subnet:
      “dbsubnetdefaultvpc91a918f5”: {
      “Type”: “AWS::RDS::DBSubnetGroup”,
      “Properties”: {
      “SubnetIds”: [
      “subnet-1ab50b43”,
      “subnet-93532cf7”
      ]
      “DBSubnetGroupDescription”: “Created from the RDS Management Console”,
      } },
    • This was successful

 

  • FAIL: Route4, Route1; Exactly one of [GatewayId, NatGatewayId, InstanceId, VpcPeeringConnectionId, NetworkInterfaceId, EgressOnlyInternetGatewayId] must be specified and not empty.
    • Attempting to remove from the JSON script, routes 4 and 1:
      “route4”: {
      “Type”: “AWS::EC2::Route”,
      “Properties”: {
      “DestinationCidrBlock”: “0.0.0.0/0”,
      “RouteTableId”: {
      “Ref”: “rtbdbb390bf”
      }}},

      “route1”: {
      “Type”: “AWS::EC2::Route”,
      “Properties”: {
      “DestinationCidrBlock”: “0.0.0.0/0”,
      “RouteTableId”: {
      “Ref”: “rtb40ad8e24”
      }}},

    • This was successful

 

  • FAIL: Route 5, Route 2; Exactly one of DestinationCidrBlock and DestinationIpv6CidrBlock must be specified and not empty.
    • Inserting “DestinationCidrBlock”: “0.0.0.0/0”, into Properties for Route2 in the JSON script.
    • Inserting “DestinationCidrBlock”: “10.0.0.0/19”, into properties for Route 5 in the JSON script (This relates to subnet cidr 1A)
    • This was successful

 

  • FAIL: Route 5, Route 2; The Gateway ID (vpce-20e01049) does not exist.
    • Attempting to delete both routes from the JSON script:
      “route2”: {
      “Type”: “AWS::EC2::Route”,
      “Properties”: {
      “DestinationCidrBlock”: “0.0.0.0/0”,
      “RouteTableId”: {
      “Ref”: “rtb40ad8e24”
      },
      “GatewayId”: “vpce-20e01049”
      }},

      “route5”: {
      “Type”: “AWS::EC2::Route”,
      “Properties”: {
      “DestinationCidrBlock”: “10.0.0.0/19”,
      “RouteTableId”: {
      “Ref”: “rtbdbb390bf”
      },
      “GatewayId”: “vpce-20e01049”
      }},

    • This was successful

 

  • FAIL: lcADDSScenario3RDGWStack1USE0PZ69GKRQRDGWLaunchConfiguration1QJ9NVFDQSTXX;
    Invalid IamInstanceProfile: AD-DS-Scenario-3-RDGWStack-1USE0PZ69GKRQ-RDGWHostProfile-CLZHHC4VKEC1

    • Attempting to delete section from JSON script (line 269) … LaunchConfiguration… object:
      “IamInstanceProfile”: “AD-DS-Scenario-3-RDGWStack-1USE0PZ69GKRQ-RDGWHostProfile-CLZHHC4VKEC1”,
    • This was successful

 

After resolving all of these errors, my CloudFormer template was able to create a stack without any rollbacks.

037 CFTemplate Complete

Although I am pleased that I managed to enable the stack to reach the status of ‘CREATE_COMPLETE’, due to the large amount of script deleted, I am uncertain as to whether my script still runs as it was originally designed.

When I run my template though the AWS template designer, the following diagram is displayed.
038 CFTemplate DesignerTemplate

This diagram does not look the same as the sample diagram found in the Microsoft Quick Start guide for ‘Scenario 3’, shown below.
Figure for Scenario 3

My next step then, is to compare the two diagrams to determine the discrepancies between my stack template and the sample template.

Adjusting the CloudFormer Template

Once the AWS CloudFormer template has been created, it needs to be run through the AWS CloudFormation ‘Create Stack’ option. This has been done by copying the CloudFormer JSON script into the Notebook++ program, and then uploading the Notebook++ file during the ‘Select Template’ portion of the Stack creation settings.

000 Upload File

Once, the template has been chosen, it requires a name, and the option of a tag. For my stacks, they have been numbered in regards to how many iterations of the script I have run through ‘Create Stack’.

010 NewReview

During the creation process of the stack, the created events can be viewed.

005 Create_In_Progress

The initial template stack contains errors that cause the creation process to rollback, and fail to complete the stack’s creation.

006 Rollback Error

The method used in removing the errors, was to find the first ‘CREATE_FAILED’ event for the stack, and attempt to solve this event failure based upon the information provided in the right-hand column of the event.

For my first CloudFormer script, came across the following problems, which I attempted to solve.

Adjusting the CF Script

  • DestinationCidrBlock Empty
    • For each event fail that referenced this error, I sourced the IPs and their respective Cidr from the initial Microsoft ‘Scenario 3’ stack creation settings, and placed them into their corresponding Routes based upon the route description.
  • Missing NetworkInterfaceID
  • Unresolved Dependencies
    • The unresolved dependencies were specific to two different routes. As I couldn’t determine a resolution to this error, I saved a copy of each of the routes, then deleted them.

These errors comprised the bulk of my event failures, and throughout each iteration of the stack, more errors would evolve. Most of these errors were derived from the network interface JSON script that I had added to the CloudFormer script. I eventually came to conclusion, with the help of others, that I should re-attempt the Microsoft Quick-Start ‘Scenario 3’ template through CloudFormation and Cloudformer, which would provide me with a clean template to work upon again.

The persuasive reason behind this action was the knowledge that this project was not intended to focus on the intricacies of the JSON script, which I had been doing in my attempts to successfully fix my first CloudFormer script.

Budget Update: 28/05/2017

This past week has also been spent on adjusting a CloudFormer template. As such, my current expectation for my billing list is that it will be quite low. The only expenses that should have occurred, are the AD-DS CloudFormation template (description link here) that I ran again, and the charge from the successful build of the CloudFormer template’s stack.

When looking at my billing list for my account, only two services have been used; EC2, and KMS. Upon further inspection, I have determined that the KMS charge is from an encryption key set in US-East region, which is not related to the AD-DS build.

The EC2 billing report is as follows:
EC2

 

I input this data, as well as last weeks data, in order to grasp a greater understanding of what charges have been incurred.

AD DS Budget Sheet

It appears that this past week has been less expensive than the one prior. This reduced charge is found within the EBS service specifics, which still contains the carry-over charges from the DinoStore volumes that I recently deleted. Hence, my first EBS budget isn’t exclusive to the AD-DS project charges and needs to be adjusted.

The adjusted AD-DS Budget Sheet is below, where the fee discrepancy has been resolved using the information from the ‘Budget Update: 14/05/2017‘, which contains my last reference to the DinoStore EBS volumes.

Budget Sheet Adjusted

I am unsure of the $0.19 difference in the General Purpose SSD (gp2) provisioned storage expense for the two dates, but may be able to verify whether this expense difference could be considered outlying with my next budget report.

 

Microsoft Active Directory Quick Start: Scenario 3

In order to fully determine how the ‘Microsoft Active Directory Quick Start: AD DS with AWS Directory Service on the AWS Cloud’ template works, I implemented the template into AWS CloudFormation’s ‘Create Stack’ option.
001 New Stack_LI

The creation of this stack involved many specific details that I hadn’t previously considered would be of importance. However, as I went through the details, I recalled seeing them within the JSON scripts that I had been altering.
003 Specify Details NS AZ_Options004 Specify Details NS EC2_Config N_D005 Specify Details NS RDGW N_D

There were a few things that had me momentarily confused. One of these was with my availability zones, as I had not realized that my account region was set to Oregon. As I live in New Zealand, the Sydney region is the most optimal for reducing latency. To change this, I needed to quit this stack creation and start it again within the Sydney region. Also, because of initial lack of realization of setting up the template within a different region, I did not understand why I could access my key pair. The revelation of being in the wrong region helped my realize that key pairs are region specific, something which  I did not know earlier.

The next step in setting up the stack was the ‘Options’, in which I created a tag for the stack, but did not change any of the permission settings as I currently don’t want to create more potential complication than necessary. This is something that I could consider adjusting once I manage to successfully recreate the CloudFormer version of this stack.
006 Options NS

Upon reviewing the reviewing the stack template to be created, and pressing ‘create’, the site responded with the following error:
010 Error from CIDR

In my ignorance of what information was required for the stack creation, I had left the Remote Desktop Gateway (RDGW) CIDR blank in the ‘Network Configuration’ section. I had also left the ‘Microsoft Active Directory’ section blank, as I didn’t fully understand why it was needed, and so had left it clear.

My first attempt to solve to the RDGW CIDR was to input the VPC CIDR into it. However, this did not work. One of my classmates suggested using the CIDR from the first public subnet, which ended up working, as it appears that the gateway required a larger mask than the one that is supplied with the VPC.
012 RDGW IP config NS

I also filled in the Microsoft Active Directory section once I realized that it required input.
011 AD Config for NS

 

Once these problems were resolved, I was able to create the stack.
018 AD_DS Stacks
This particular template created four stacks; an AD stack, a VPC stack, an RDGW stack, and a general stack. Each of which, implemented certain AWS services.

The RDGW stack had the following outputs:
022 RDGW Stack Outputs

 

The AD stack had the following outputs:
023 AD Stack Overview

 

The VPC stack had the following outputs:
024 VPC Stack Outputs

 

The general stack had no outputs:
025 3S Stack Outputs

 

The outputs from the stacks are important to know as they can be selected in CloudFormer, but don’t necessarily have a tag attached, which can make them hard to distinguish from any other service objects that still exist within the AWS system. It is also important because not all of them exist on the free tier, which means that if the stacks are left running, a large fee can quickly accrue. The simplest way to stop the fees is to delete the stacks as that deletes all of the service objects involved. This is not always the optimal choice, but it is the best one for me once I’ve mapped the stacks through CloudFormer.

AD in CloudFormation: JSON files

When starting up scenario 3 (mentioned in my previous blog), the CloudFormation service offers the option of ‘View/Edit template in Designer’. This provides me the JSON script for the template, and a physical diagram.

CF JSON and PhysMod

My first attempt in understanding and manipulating the template was by playing with the JSON scripts. I was playing around with the template, which contained links between different stacks. This made it harder to adjust the code as I needed to constantly determine where the script was pointing.

In the end, I was unable to modify the code and keep it in working order. However, this endeavor did help me gain an understanding of the code, for its format, and what AWS services it contains.

This information should provide me greater understanding in how to adjust the template that I will gain through my next method of using CloudFormer.

Introduction to Active Directory Domain Services

This next project involves using AWS’s CloudFormation service in building an active directory (AD), and also deploying and automating it.

Amazon CloudFormation service is described as “a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications”.
Hence, it is an applicable service for deploying and automating an AD. –What is AWS CloudFormation

The specific CloudFormation template that will be used for this project is the ‘Microsoft Active Directory Quick Start: AD DS with AWS Directory Service on the AWS Cloud’, which is scenario 3.
Quick Start Options

The template is designed to provide the following architecture:
Figure for Scenario 3

The purpose for this project is to redesign the template so that it is simplified, improved, optimized, and personalized.

Budget Update: 17/05/2017

For the past week I have been working with AWS CloudFormation to create and automate an Active Directory using scenario 3 from the following AWS manual.

This scenario involves the creation of stacks that utilize Amazon services from their enterprise level, which results in this being a potentially very expensive endeavor.

After running the stacks for a few hours, my billings showed the following EC2 charges:
EC2 Billing

My conclusion from this, was that I needed to create a budget alarm to ensure that I remain aware of the involved expenditure for this project.

AD DS Alarm

The current limit for the budget is only $10 because my intention is not to try and stay within this budget, but to keep it small enough that I remain conscious of the expenditure. The purpose for this is to provide myself with minimal cost involvement in comprehending the cost value associated with the stack.

Introduction to AWS CloudFormation

Introduction and Aim
The purpose of this lab is to use an Amazon EC2 instance and install WordPress with a local MySQL database. QwikLabs states that AWS CloudFormation ‘gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.’

 

Goals

  • Create a stack using an AWS CloudFormation template
  • Monitor the progress of the stack creation
  • Use the stack resources
  • Clean up when the stack is no longer required

 

Create a stack
In this section, I create a stack  from an AWS CloudFormation template.
InkedCreate Stack_LI

CloudFormation is one of the services found in the AWS management console. In the service, I can ‘Create Stack’, selecting the ‘WordPress blog’ template.
The details are as follows:
Name: MyWPTestStack
DBPassword: Pa55word
DBRootPassword: Pa55word1
DBUser: AWSQLStudent
Specifying Details

The lab script makes mention here that ‘the same WordPress template contains an input parameter, KeyName, which specifies the EC2 key pair for the Amazon Ec2 instance that is declared in the template. An Amazon key pair has been created for you.’

As I’m only following the lab script, not actually completing the lab through QwikLabs, I don’t have a pre-made template. However, I do have access to creating EC2 instances, alongside the ones I’ve already created, and I already have a key pair.

In the KeyName drop down on the Details page, I select the key pair that I’ve already created.

The automatically filled parameters are kept on their default settings, and no ‘Tags’ or ‘Advanced Options’ settings are changed, so all that is left to do is create the instance.

Stack Review

Monitoring stack creation
The AWS service for CloudFormation monitors the progress of the stack’s creation. Whilst being created, the status will be CREATE_IN_PROGRESS. Once finished, the status notification will show CREATE_COMPLETE.

This slideshow requires JavaScript.

 

 

Using the stack
The WordPress installation still need to be completed. This is done by clicking on the outputs tab, and using the hyperlink located on the page.
Outputs WP
Once the installation is complete, the WordPress dashboard appears. From here, customization and blog posts can happen.

This slideshow requires JavaScript.

 

Deleting the stack.
Deleting the stack involves selecting the stack to be deleted, under ‘Actions’ pressing ‘Delete Stack’, and then confirming the deletion process.
Delete Stack

During the process, the stack status changes to DELETE_IN_PROGRESS. When a stack is deleted, all of the resources associated with the stack will also be deleted.
DELETE_IN_PROGRESS

 

Conclusion
By following this lab, I’ve manged to learn how to use CloudFormation in creating a stack, and install a WordPress template.  The implementation of stacks appear to be very useful for running applications, though I would be interested in comparing it to the AWS Lambda service.