Adjusting the CloudFormer Template

Once the AWS CloudFormer template has been created, it needs to be run through the AWS CloudFormation ‘Create Stack’ option. This has been done by copying the CloudFormer JSON script into the Notebook++ program, and then uploading the Notebook++ file during the ‘Select Template’ portion of the Stack creation settings.

000 Upload File

Once, the template has been chosen, it requires a name, and the option of a tag. For my stacks, they have been numbered in regards to how many iterations of the script I have run through ‘Create Stack’.

010 NewReview

During the creation process of the stack, the created events can be viewed.

005 Create_In_Progress

The initial template stack contains errors that cause the creation process to rollback, and fail to complete the stack’s creation.

006 Rollback Error

The method used in removing the errors, was to find the first ‘CREATE_FAILED’ event for the stack, and attempt to solve this event failure based upon the information provided in the right-hand column of the event.

For my first CloudFormer script, came across the following problems, which I attempted to solve.

Adjusting the CF Script

  • DestinationCidrBlock Empty
    • For each event fail that referenced this error, I sourced the IPs and their respective Cidr from the initial Microsoft ‘Scenario 3’ stack creation settings, and placed them into their corresponding Routes based upon the route description.
  • Missing NetworkInterfaceID
  • Unresolved Dependencies
    • The unresolved dependencies were specific to two different routes. As I couldn’t determine a resolution to this error, I saved a copy of each of the routes, then deleted them.

These errors comprised the bulk of my event failures, and throughout each iteration of the stack, more errors would evolve. Most of these errors were derived from the network interface JSON script that I had added to the CloudFormer script. I eventually came to conclusion, with the help of others, that I should re-attempt the Microsoft Quick-Start ‘Scenario 3’ template through CloudFormation and Cloudformer, which would provide me with a clean template to work upon again.

The persuasive reason behind this action was the knowledge that this project was not intended to focus on the intricacies of the JSON script, which I had been doing in my attempts to successfully fix my first CloudFormer script.

Budget Update: 28/05/2017

This past week has also been spent on adjusting a CloudFormer template. As such, my current expectation for my billing list is that it will be quite low. The only expenses that should have occurred, are the AD-DS CloudFormation template (description link here) that I ran again, and the charge from the successful build of the CloudFormer template’s stack.

When looking at my billing list for my account, only two services have been used; EC2, and KMS. Upon further inspection, I have determined that the KMS charge is from an encryption key set in US-East region, which is not related to the AD-DS build.

The EC2 billing report is as follows:
EC2

 

I input this data, as well as last weeks data, in order to grasp a greater understanding of what charges have been incurred.

AD DS Budget Sheet

It appears that this past week has been less expensive than the one prior. This reduced charge is found within the EBS service specifics, which still contains the carry-over charges from the DinoStore volumes that I recently deleted. Hence, my first EBS budget isn’t exclusive to the AD-DS project charges and needs to be adjusted.

The adjusted AD-DS Budget Sheet is below, where the fee discrepancy has been resolved using the information from the ‘Budget Update: 14/05/2017‘, which contains my last reference to the DinoStore EBS volumes.

Budget Sheet Adjusted

I am unsure of the $0.19 difference in the General Purpose SSD (gp2) provisioned storage expense for the two dates, but may be able to verify whether this expense difference could be considered outlying with my next budget report.

 

Microsoft Active Directory Quick Start: Scenario 3

In order to fully determine how the ‘Microsoft Active Directory Quick Start: AD DS with AWS Directory Service on the AWS Cloud’ template works, I implemented the template into AWS CloudFormation’s ‘Create Stack’ option.
001 New Stack_LI

The creation of this stack involved many specific details that I hadn’t previously considered would be of importance. However, as I went through the details, I recalled seeing them within the JSON scripts that I had been altering.
003 Specify Details NS AZ_Options004 Specify Details NS EC2_Config N_D005 Specify Details NS RDGW N_D

There were a few things that had me momentarily confused. One of these was with my availability zones, as I had not realized that my account region was set to Oregon. As I live in New Zealand, the Sydney region is the most optimal for reducing latency. To change this, I needed to quit this stack creation and start it again within the Sydney region. Also, because of initial lack of realization of setting up the template within a different region, I did not understand why I could access my key pair. The revelation of being in the wrong region helped my realize that key pairs are region specific, something which  I did not know earlier.

The next step in setting up the stack was the ‘Options’, in which I created a tag for the stack, but did not change any of the permission settings as I currently don’t want to create more potential complication than necessary. This is something that I could consider adjusting once I manage to successfully recreate the CloudFormer version of this stack.
006 Options NS

Upon reviewing the reviewing the stack template to be created, and pressing ‘create’, the site responded with the following error:
010 Error from CIDR

In my ignorance of what information was required for the stack creation, I had left the Remote Desktop Gateway (RDGW) CIDR blank in the ‘Network Configuration’ section. I had also left the ‘Microsoft Active Directory’ section blank, as I didn’t fully understand why it was needed, and so had left it clear.

My first attempt to solve to the RDGW CIDR was to input the VPC CIDR into it. However, this did not work. One of my classmates suggested using the CIDR from the first public subnet, which ended up working, as it appears that the gateway required a larger mask than the one that is supplied with the VPC.
012 RDGW IP config NS

I also filled in the Microsoft Active Directory section once I realized that it required input.
011 AD Config for NS

 

Once these problems were resolved, I was able to create the stack.
018 AD_DS Stacks
This particular template created four stacks; an AD stack, a VPC stack, an RDGW stack, and a general stack. Each of which, implemented certain AWS services.

The RDGW stack had the following outputs:
022 RDGW Stack Outputs

 

The AD stack had the following outputs:
023 AD Stack Overview

 

The VPC stack had the following outputs:
024 VPC Stack Outputs

 

The general stack had no outputs:
025 3S Stack Outputs

 

The outputs from the stacks are important to know as they can be selected in CloudFormer, but don’t necessarily have a tag attached, which can make them hard to distinguish from any other service objects that still exist within the AWS system. It is also important because not all of them exist on the free tier, which means that if the stacks are left running, a large fee can quickly accrue. The simplest way to stop the fees is to delete the stacks as that deletes all of the service objects involved. This is not always the optimal choice, but it is the best one for me once I’ve mapped the stacks through CloudFormer.

AD in CloudFormation: JSON files

When starting up scenario 3 (mentioned in my previous blog), the CloudFormation service offers the option of ‘View/Edit template in Designer’. This provides me the JSON script for the template, and a physical diagram.

CF JSON and PhysMod

My first attempt in understanding and manipulating the template was by playing with the JSON scripts. I was playing around with the template, which contained links between different stacks. This made it harder to adjust the code as I needed to constantly determine where the script was pointing.

In the end, I was unable to modify the code and keep it in working order. However, this endeavor did help me gain an understanding of the code, for its format, and what AWS services it contains.

This information should provide me greater understanding in how to adjust the template that I will gain through my next method of using CloudFormer.

Introduction to Active Directory Domain Services

This next project involves using AWS’s CloudFormation service in building an active directory (AD), and also deploying and automating it.

Amazon CloudFormation service is described as “a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications”.
Hence, it is an applicable service for deploying and automating an AD. –What is AWS CloudFormation

The specific CloudFormation template that will be used for this project is the ‘Microsoft Active Directory Quick Start: AD DS with AWS Directory Service on the AWS Cloud’, which is scenario 3.
Quick Start Options

The template is designed to provide the following architecture:
Figure for Scenario 3

The purpose for this project is to redesign the template so that it is simplified, improved, optimized, and personalized.

Budget Update: 17/05/2017

For the past week I have been working with AWS CloudFormation to create and automate an Active Directory using scenario 3 from the following AWS manual.

This scenario involves the creation of stacks that utilize Amazon services from their enterprise level, which results in this being a potentially very expensive endeavor.

After running the stacks for a few hours, my billings showed the following EC2 charges:
EC2 Billing

My conclusion from this, was that I needed to create a budget alarm to ensure that I remain aware of the involved expenditure for this project.

AD DS Alarm

The current limit for the budget is only $10 because my intention is not to try and stay within this budget, but to keep it small enough that I remain conscious of the expenditure. The purpose for this is to provide myself with minimal cost involvement in comprehending the cost value associated with the stack.

Budget Update: 14/05/2017

At close on midnight last night, I received an AWS notification email, telling me that my EC2 budget had passed one of its limits. Since my last budget post, my activity with AWS has been within the CloudFormation services, which is free-tier.
Budget Alarm

 

To understand why I was being charged, I checked upon billing info. The results showed that I was still being billed for EBS services.
14_05_17 Bill Report

 

My conclusion was that I had not completely deleted my EBS volumes during my last attempt. In checking the EC2 service, I was proven right.
EBS Volumes

 

This time, I made sure to delete my EBS volumes, and stop the unnecessary credit charge.
Deleting EBS Volumes

 

My next budget update should only show purposeful charge, which I have initiated.

 

 

Budget Update: 08/05/2017

For the past week, ever since the DinoStore project ended, my AWS account hasn’t been used.

As it is a new month, my billing log has also rolled over.
Billing Total
The charge of $0.14 credits are from two services; EC2 and KMS.

The EC2 charge, as seen in the screenshot below, was from the EBS service within EC2. The instances that I had been creating for my DinoStore had automated volumes within the EBS service system.
EC2 only 1

When I had shut down the rest of my DinoStore services, I had forgotten about EBS, which is why I have been charged even though I am no longer working upon the DinoStore project.

 

The KMS charge is a charge for which I am unsure of its origin.
KMS only 1
A large factor in my confusion is that the KMS specifies a region-based charge, but when I trace it back into the IAM service, IAM is not region specific, so I’m left unsure as to how I am supposed to determine, and reduce this charge.

As of current, my guess is that may be due to something occurring each time I log into AWS, but I am still very much unsure.

I can’t seem to find any relating AWS documentation, so my current plan is to check with my classmates to determine whether they have the same charge on their Billing Info page.

 

 

 

 

Billing Report: Billing Invoice

On May the 3rd, I received this email from AWS:
InkedAWS bill_LI

This slightly concerned me as this was a direct charge to my card rather than my credit.

My first response was to go to the AWS Billing page and determine the reason for this charge. The Billing dashboard showed an amount $0.64 expenditure had occurred, and I, being unsure as to what this charge specified, checked my credits to determine whether the amount had been removed from there.

Credits Remaining

My credit page showed a fee charge similar to what I had calculated in my spreadsheet, of $16.17 USD credits.
Spreadsheet

This boosted my confidence in understanding my credit usage and fees, but didn’t resolve the reason why my card was being charged.

The next response was to click on the graph which showed my expenditure, and see if that helped. This lead me to the Cost Explorer option in the Billing Console, and showed me this information:
Billing Diagram

 

From this, I can see that Budget and Tax appear to make up the bulk of the costs.

By refining the Time range, I was able to preview my costs more clearly:
Cost Explorer

This one clearly shows that the Budget Service cost me $0.56 and Tax was $0.08, which comes to $0.64 total., which equals the amount that I was directly charged.

My conclusion from this is that not every service is charged to my credit, and that although the budgets are fantastic for warning me of my credit expenditure, I should keep in mind that they also come with a cost.

Budget Update: 30/04/2017

The billing report in AWS only displays expenditure during the current month. For this month, the three main causes of expense have been the services; RDS, EC2, and KMS.

More detailed billing logs for each service are shown below.

This slideshow requires JavaScript.

When input into my AWS spreadsheet, I get the following results:

SpreadSheet Budget

 Spreadsheet documenting AWS billing services

Spreadsheet Percentage growth budget

 Spreadsheet calculating percentage growth between each budget update

The percentage growth between this budget update and the last one has dropped, which is indicative of me reaching the end of the DinoStore project and hence, using those services less frequently.