In the last post, I covered the basics of EC2, including how instances are billed, which components make up an instance and then how to create and connect to your instance. In this post, we’re going to take a look at some of the more advanced aspects of EC2 - topics such as Auto Scaling, using Systems Manager to more effectively manage your instances and how to gather information about your instances using the built in metadata function.
Auto Scaling
Imagine you have an application that runs on a single EC2 instance. Most of the time, the number of people using that application is fairly low, but at certain times of the day (or on certain days), the number of users increases dramatically. How do you deal with this? You could “overprovision” the number of instances you use to support that application (i.e having the appropriate number of instances required to support the higher user count), but that means you are paying for those instances all of the time - even when they aren’t really being used. A better way to solve this problem would be to use the EC2 Auto Scaling feature.
The basic idea behind Auto Scaling is that the AWS platform will attempt to keep the number of provisioned instances at a “desired” level. If you specify that your desired number of instances is five, then EC2 will add instances if necessary to meet that requirement. So if one out of the five instances should fail, the platform will automatically add another instance to bring the total number back up to five.
In addition, Auto Scaling will add or remove instances automatically whenever a particular condition is met. So for example, you might define a condition that states that if the CPU utilisation of your instances breaches a threshold (let’s say 70%), Auto Scaling will begin to create more instances for you.
Launch Templates & Configurations
The first thing you’ll need to do when setting up an Auto Scaling group is define either a Launch Template or Launch Configuration. These are similar in that they both allow you to define how the instances within your Auto Scaling group should look, but Launch Templates are newer and slightly more flexible, allowing for updating and versioning. A Launch Template or Launch Configuration is simply a template from which the instances in your Auto Scaling group will be created. What does a Launch Template or Configuration consist of? Well, mostly the same kind of parameters as if you were defining an instance directly - you will need to specify parameters such as the AMI to use, an instance type (e.g. T2.Micro), the network interface parameters and storage details. Here’s an example of how a Launch Template is created:
Auto Scaling Groups
Once you have your Launch Template / Configuration, you can go on to create an Auto Scaling Group. This allows you to specify parameters such as how many instances are desired, which specific VPC should be used for the deployment and any scaling policies and notifications you may wish to set up. A scaling policy might state that you want to increase the amount of EC2 instances whenever a specific condition is met (such as CPU utilisation hitting a certain percentage). The AWS platform will monitor this condition inside your Auto Scaling group and then attempt to increase the number of instances once this condition has been met. This screenshot shows a simple scaling policy that will try and add additional instances once the CPU utilisation exceeds 50%:
It’s also possible to add “step” policies, which allow you to specify additional steps for scaling - for example, step 1 could add two additional instances when the CPU utilisation is between 30 and 50%. Step 2 could add two more instances when the CPU utilisation is between 50% and infinity.
Systems Manager
Next up, let’s take a look at what EC2 Systems Manager has to offer. Systems Manager was introduced towards the end of 2017 - it isn’t actually one feature, but more a collection of features that are designed to simplify the management of and provide greater insight and visibility into your cloud environment. One of my favourite features of Systems Manager is the ability to group AWS resources together using the Resource Groups feature.
Resource Groups
What is a resource in AWS? A resource is pretty much anything you can work with, such as an EC2 instance, an S3 bucket, a VPC and so on. As your environment grows, it’s likely that the number of resources that you are using will grow over time. Without a way to organise these resources, things may start to get a little confusing and difficult to manage.
Enter Resource Groups. The idea behind resource groups is simple - provide a way to logically organise the resources in your AWS account and allow you to perform actions on those resources together. How do resources “get” into a resource group? The primary method for allocating resources to resource groups is by using tags. A tag is essentially a label; a way to categorise resources. Tags consist of a key value pair - so for example, I could tag all resources that I own with something like this:
Owner = Adam
OK, that’s easy enough to understand, but how do we use tags to group our resources together into resource groups? Well let’s say I have an application called “App10”, which uses a number of EC2 instances. I’d like to manage those instances together if possible, so I might apply tags to all of the EC2 instances that are used for the “App10” app as follows:
Note that I also have one other instance that is used for a completely separate app (“App20”), but I don’t want to include that in the group and so have given it a different tag.
Now, I click on ‘Resource Groups’ at the top of the AWS console - I create a new group and then use a resource type of “AWS::EC2::Instance” and a tag of “App: App20”, as shown in the following screen shot:
You can see from this that four instances have been identified and will therefore be members of this resource group. Further down on the screen, I can name the resource group, as well as apply a tag to the resource group itself (resource groups can be nested).
I could also have created this resource group from the AWS CLI, for example:
So, now that we have a resource group in place containing a number of resources, what can we actually do with it? Well, at this point, we can start looking at the other features available within Systems Manager - first up, Insights.
Insights
The Insights functionality found Systems Manager is focused primarily on the inventory and compliance within your environment. By clicking on ‘Inventory’ within the Systems Manager console, you can quickly get a view of the managed instances you are running (a managed instance is simply an EC2 instance that has the appropriate software to allow it to be managed by Systems Manager), as well as the operating systems and OS versions, plus the applications running on those instances:
As you can see, it’s possible to filter by resource group or other criteria, such as tags and inventory type. Insights also has a built-in compliance view, which lets you easily look at patch management and configuration rules that you might want to apply to your environment.
Run Command
SSH’ing into machines is so old hat isn’t it? What if you didn’t have to bother with that and instead had the ability to remotely run commands on your instances? Step forward, Run Command. Here’s an example - I want to run a shell script on each of my managed instances. Rather than using SSH to get into each machine individually and running the script, I can use Run Command to do this instead:
Once I click the ‘Run’ button, the command executes on each instance:
There are a number of other features within Systems Manager that I won’t cover here, including Session Manager (a feature that lets you gain access to the shell of an instance through the browser), Patch Manager and State Manager (allows you to manage the state of your instances, for example anti-virus or firewall settings).
Instance Metadata
The EC2 instances that you run on AWS include a large number of configuration items and settings - for example:
Which AMI (Amazon Machine Image) was used to create the instance?
What role is assigned to instance currently?
What IP addresses are associated with the instance?
Which Availability Zone is the instance running in?
There are lots of others, but the point is that it would be handy if there was a way to gain access to this information from within the instance itself. This is where Instance Metadata comes in. Let’s see how this works.
In the following example, I’m SSH’d into one of my Linux EC2 instances. To access the metadata for the instance, you simply need to make an HTTP request to a specific end point in the form of http://169.254.169.254/latest/meta-data/. So if for example, I want to know the host name of my instance, I can use the Curl tool to make the request:
The Elastic Compute Cloud (EC2) service is one of the best known and widely used services that AWS offers. What it provides is fundamentally quite simple: the ability to provision virtual machines (known as instances in AWS speak) on demand. In contrast to many of the ‘managed’ services on offer, with EC2 the user has complete control over the provisioned instances and is responsible for all software installation, configuration, patching and so on.
One of the first things to understand about EC2 is how you are billed for it; let’s take a look.
EC2 Billing
There are three main ways to pay for EC2 instances; which one you choose really depends upon how you are using the service, whether you are willing to make a spending commitment in return for lower prices and so on.
On-Demand Instances
The on-demand pricing model essentially provides you with a ‘pay-as-you-go’ type of approach - you simply spin up your instance and then pay for the amount of time it is running. And that’s really it - you don’t need to make any kind of commitment, up front payments, etc.
So when would you use on-demand instances? Think about any kind of short term requirement such as testing, development of a new app, or learning about the platform. In those cases, users probably don’t want to make any monetary commitment up front as they won’t know how much time they need their instances for.
So how much does it cost to run an on-demand instance? That depends on many different factors, such as the instance size (see later section), the operating system chosen (Linux is normally going to be cheaper than Windows) and perhaps even the AWS region you are running the instance in.
It used to be the case that EC2 instances were billed per hour, however this changed some time back. Nowadays, instances are billed per second (albeit with a 60 second minimum). So if I spin up an instance and use it only for 30 seconds, I’ll be billed for one minute. This is also the case for spot and reserved instances, which needs us nicely into the next section.
Reserved Instances
Let’s say you know that you will need to run a certain number of instances for the foreseeable future - in that case, on-demand pricing might not make too much sense. A better option would be to make use of EC2 reserved instances. The idea here is that you make an up front payment to AWS for the instances you want to run for a period of time (such as 1 or 3 years) - by doing so, you can enjoy significant discounts (sometimes up to 75%) over on-demand pricing.
One other advantage of reserved instances is that you can also reserve a certain amount of capacity in a specific availability zone - for example, if I select the AZ ‘eu-west-2a’ when provisioning my reserved instances, I have some additional peace of mind in that I know there will be capacity available for my instances.
There are a few types of RI available depending on how you might wish to use them - more information is available on the RI pricing page here.
Spot Instances
Imagine you are responsible for running a cloud platform such as AWS. Inevitably, there are going to be parts of the infrastructure that - at times - are not necessarily being used, but that need to be kept available to provide capacity to users if required. It might therefore make sense for you to offer that unused infrastructure out to users at a lower price, on the proviso that you may need that infrastructure back at any point with very little notice.
What I’ve described above is essentially how spot instances work - they provide the ability to provision EC2 instances at a lower price than if using the on-demand pricing model. Here’s how spot instances work:
The user creates a spot instance request. This request includes (among other things) the maximum price that the user wishes to pay for their instances.
AWS publishes a list of spot prices for each type of instance. You can find the up to date spot pricing on this page.
If the maximum bid price in your spot instance request is higher than the current spot price for the instance in question, your instance will be provisioned (assuming there is capacity available of course).
If the maximum bid price in your request is lower than the current spot price, the instance will not be provisioned, or in the case of currently running instances, those instances will be terminated.
What could cause your maximum bid price to be lower than the current spot price? Well, the spot price can move (depending on a number of factors such as demand and capacity), so it is possible that the spot price could rise above your maximum bid price at some point. To help you set your maximum price, you can view the spot price history using the AWS console or command line.
At first glance, the thought of provisioning instances that AWS can whip out from under you at any moment might not sound that appealing. However, there are some valid use cases - think about extremely fault tolerant applications such as Hadoop or image rendering, as well as in cases where some downtime is acceptable (test servers for example).
Components of an Instance
What is an instance made up of? If you think about an ‘on-premises’ server, that would typically be built using a number of components, including processor, memory, disk, network interfaces and so on. An instance or virtual machine running in the cloud should therefore be thought of in a similar kind of way - an AWS instance is not a single ‘thing’, but instead consists of a number of components, all of which are extremely important for the proper deployment and operation of that instance. Let’s have a look at these components:
Instance
Number 1 on the list is the instance itself - obviously without this, there isn’t much point in having the additional components mentioned below. An instance is the primary method for providing virtualised computing power in AWS. An instance will have a specific configuration, such as the operating system, the amount of CPU and memory, as well as the state of the instance (e.g. running or stopped). There are a huge variety of instance sizes available to cover almost every conceivable use case, ranging from very small (e.g. 2 vCPUs and 0.5Gb of memory) to huge, high performance machines with terabytes of memory and over 100 vCPUs.
Amazon Machine Images (AMIs)
If the instance size discussed in the last section defines the ‘hardware’ configuration of the instance (CPU, memory, etc), what defines the software configuration of the instance? For that, we look to Amazon Machine Images, or AMIs for short. So what does an AMI define exactly? First and foremost, it defines the operating system that will run on the instance. A huge number of AMIs are available - if you just want an instance that runs Ubuntu, there’s an AMI for that. Prefer Windows? No problem, there are multiple AMIs available for that too. There’s even an entire marketplace where you can find AMIs from 3rd party vendors, to run everything from firewall software to computational molecular biology machines.
But what if you can’t find - among the tens of thousands of AMIs available - the exact configuration you need? Well that’s not an issue either - you simply create your own. After customising the instance, creating a new AMI from that instance is a trivial matter that involves EC2 creating a snapshot, from which your new instances will be created in the future. Once you’ve created your AMIs, you can make them public, or keep them private for your own use only.
One other important thing an AMI defines is the disk configuration of your instance - specifically, whether the root device of your instance uses EBS or instance store. These are new terms, so that brings me nicely to the next section: disks.
Disks
Every instance that you provision requires a disk to be attached to it. AWS provides two main methods for how you attach those disks: Instance Store or Elastic Block Store (EBS) volumes. What’s the difference?
Instance Store Based Disks
With an instance store, the underlying host computer that your EC2 instance is running on has disks physically attached to it - your instance disk is provisioned on this directly attached drive, as shown below.
Sounds good, but what happens if the host computer your instance is running on fails? In that case, any data you have stored on the disk will be lost. The same is true if the instance is stopped or terminated. Because of this, it’s quite unwise to store any data that needs to be kept permanently on an instance store based disk - instance stores are really only suitable for temporary data that you don’t mind losing.
If you have data that must be retained on a longer term basis, you need to look at EBS based volumes instead.
Elastic Block Store (EBS) Volumes
An EBS volume is a block based storage device that you can attach to your EC2 instance. The key difference between an EBS volume and an instance store is that the EBS volume does not live on the underlying host machine, but instead is independent from the host. This means that the instance can be stopped or terminated, but the data on the EBS volume will persist.
EBS volumes are also highly available - they are replicated within an Availability Zone (AZ) to avoid the situation where hardware failure results in loss of data.
I’ll cover EBS in some more detail in a subsequent post.
Network Interfaces
Just as a physical server needs a network interface card (NIC) in order to connect to a network, an instance must also have some means of connecting to the networking environment. In AWS, this is achieved through an Elastic Network Interface, or ENI for short. An ENI is simply a virtual network card that attaches to an instance and is used to connect to a Virtual Private Cloud (VPC - more on this in the networking post later on).
An ENI will typically have one or more IP addresses associated with it - depending on requirements, this might include one or more private IPv4 addresses, a public IP address, an Elastic IP address (this is essentially a static IP address that remains in your account and is available to be allocated to instances / ENIs), or a number of IPv6 addresses.
Note also that you can attach more than one ENI to an instance - how many you can attach is dictated by the instance type (e.g. c1.medium, c4.large and so on). Why would you want to do this? One reason is that you might want to separate the management traffic to your instances from the production traffic. You could achieve this by attaching two ENIs - one could attach to a public facing subnet for production traffic, with the other ENI attaching to a separate, private subnet.
Instance Types
Physical, on-premises servers come in all sorts of different configurations, with varying amounts of CPU and memory; AWS instances are no different. There are a wide variety of instance types available - an instance type defines the CPU performance, amount of memory, storage options and networking performance.
As there are such a huge array of instance types available, there’s no way I can list every one out here. However, the instance types are grouped together into a number of categories - currently these are as follows:
General Purpose
CPU Optimised (with comparatively high CPU performance).
Memory optimised (with higher amounts of memory - typically used for memory intensive applications such as in-memory databases).
Accelerated Computing - graphics and GPU instances.
Storage Optimised - with high performance local storage
New instance types are being added regularly - for the most up to date information, have a look at this page.
Creating an Instance
OK, we know now what components make up an instance. How do we actually create and run one? Running an instance is a fairly simple process that can easily be performed from the AWS console, from the CLI or using an infrastructure as code tool as such as CloudFormation.
The first thing you’ll need to do when running an instance is to select the AMI you want to use:
In the above screenshot, you can see a couple of Amazon Linux images, plus a Red Hat image, but as stated in the section above, there are a huge amount of AMIs available. Next, you’ll choose the instance type (e.g. T2, M4 and so on) depending on the computing power you need.
On the next screen, you have a few configuration choices. From here, you can choose the number of instances to launch, whether to purchase spot instances and the VPC / subnet you wish to deploy your instances into. You can also assign an IAM role to your instance at this point (for example, if you want your instance to have access to S3, you might wish to assign a role with permissions that allow this).
Next, you’ll select the storage configuration for your instance (EBS, instance store, etc), followed by the tag configuration (tagging your instances is a good idea) and finally the security group configuration (I’ll cover security groups in the section on networking). And that’s it! At this point, you can hit the ‘Create’ button and your instance will spin up in one or two minutes.
Connecting to your Instance
Just before you launch your instance, a dialog box will pop up asking you to either create a new key pair, or to select an existing key pair:
You’re going to need this key pair in order to access you instance using SSH, so make sure you create this key pair and download it at this point as you won’t get the chance again. Once downloaded, you need to change the permissions on the .pem file as the default permissions are not restrictive enough:
chmod 400 MyEC2KeyPair.pem
Once the instance is ready, you can SSH into it using the following command:
ssh ec2-user@\<public-ip\>-i\<pem file\>
OK, we’ve covered a lot in this post - how EC2 instances are billed, the components of an instance, how to create an instance and then how to connect to it. In the next section, I’m going to look at a few more advanced aspects of EC2, such as Auto Scaling, Systems Manager and gathering information about your instances using the built in metadata functionality. See you there!
Identity & Access Management (IAM) is one of the key building blocks of AWS - before jumping in to the other services, it’s important to have a basic understanding of how IAM works, hence the reason for covering it fairly early on in this series.
So what exactly is IAM and how is it used? At its core, IAM is a service that provides the ability to control access to resources in AWS. What kind of entities would require access to AWS resources? Well, users and groups are the obvious ones, but IAM also provides the ability to allow services in AWS to access other services.
IAM Users and Groups
Let’s start with the basics: at the risk of stating the obvious, IAM provides the ability to define users and the groups those users are members of. I also don’t think you’ll be too surprised to hear that IAM allows you to grant certain levels of access to those users and groups. So far, so good. But let’s take a look at some of the more interesting aspects of users and groups, beginning with the root credentials for your account.
Root User Credentials
When you first create an AWS account, an identity is created for you called the root user - this is the email address and password you used when creating the account. These root user credentials have full, unrestricted access to all aspects of your AWS account. Now of course, you could quite happily use these root credentials to access your account on a day to day basis - however, alarm bells should start ringing at this point. Once the initial setup is complete, do you really need unrestricted access into your account? Probably not - therefore it is very very strongly recommended that you do not use the root user credentials for every day access to your AWS account (even for administrative tasks). You should also set up Multi-Factor Authentication (MFA) on the root account to provide even more protection against misuse of these credentials.
Helpfully, the AWS console is aware of these recommendations and tries to save you from yourself. Logging into the IAM area of the AWS console, you’ll be presented with a page which urges you to a) delete your root access keys and b) set up MFA on the root account:
So if we shouldn’t use the root credentials for admin tasks, what should we use instead? Easy: we set up a new IAM user, assign the required administrative privileges to that user and then use that new user for the majority of administrative tasks.
Principals
An user is just one example of a principal. A principal is simply an entity that can make a request to a resource in AWS. So our users that we create in IAM are principals. Our root user that we created when setting up our account is a principal. What else qualifies as a principal? Well we haven’t covered them yet but roles, federated users and applications are all principals too.
One thing that isn’t a principal is a group - a group is really just a way of providing access to multiple users at once, so it can’t be considered a principal in its own right.
Passwords, Access Key IDs and Secret Access Keys
When you create a new user in IAM, you have a decision to make about how you will interact with AWS: will you use the AWS console, will you be using programmatic access (CLI, API, etc) or will you be doing both? The choices you make here determine whether you will be issued with a password, an access key ID & secret access key, or possibly all of those. Let’s look at these options in a bit more detail.
If you choose ‘AWS management console access’ when creating a user, you will be issued with a password. You’ll use that password to access the console - yep, just a plain old password, nothing out of the ordinary there. If however you check the box named ‘programmatic access’, upon creation of the user you’ll be issued with an Access Key ID and a Secret Access Key. These credentials are used for accessing AWS programmatically, for example using the AWS CLI. It’s quite important that you make a note of these credentials when they are displayed here - the reason for this is that they will never be shown to you again. Like, ever. If you lose these credentials, you’ll have to create a new access key ID / secret access key pair.
Note that you can’t use the password for programmatic access, nor can you use the access key ID for console access.
IAM Policies
When you create a user in IAM, that user doesn’t necessarily have access to any resources - in fact, the default is to not assign any permissions to that user. A user without access to anything probably isn’t going to be all that useful to you, so how do we assign permissions to our user? The answer is IAM Policies.
An IAM policy is really just a way of defining what actions can be taken for a given resource. Here’s a really simple example of an IAM policy:
Hopefully it’s pretty easy to see what is going on here; this policy is allowing all actions (*) to be performed on any resource (*). In fact the JSON shown above defines the built in ‘AdministratorAccess’ policy within IAM. OK, that was easy, but can you guess what this next policy does?
This policy gives the users or groups to which it has been assigned the ability to perform only a couple of operations: S3 Get and List. So this essentially grants read only access to the S3 service.
Identity Policies, Resource Policies and Trust Policies
Now that you understand what a policy is, let’s make it slightly more complicated by introducing a few different types of policy.
First up is the Identity-Based Policy - this is probably the simplest policy type to understand as it involves applying the policy directly to an IAM identity. In other words, you assign the policy to a user, group or role and that identity will assume the permissions specified in the policy.
Next, we have Resource-Based Policies. The idea here is that the policy is applied to the resource, rather than the identity. For example, let’s say I have an S3 bucket (we’ll cover exactly what this is later on, but for now, think of this as an area of storage reserved for you in AWS). I want to apply a policy to this S3 bucket that allows only certain principals (i.e. users, groups or roles) to have a desired level of access. I can use a resource based policy to achieve this.
Finally, we have Trust-Based Policies. This policy type is attached to a role and defines which principals can actually ‘assume’ that role.
I’ve mentioned roles a few times already in this post, so at this point I probably ought to explain what they are, which leads me nicely into the next section.
Roles
Imagine you have an application running on an EC2 instance in AWS. That application needs to occasionally write some information to an S3 bucket, but requires credentials to do so. In this scenario, you could place the credentials on the EC2 instance, but this isn’t a great solution longer term: it’s not all that secure and should any credentials need to be revoked, all the EC2 instances you are running would need to be updated with the new ones. A better solution in this case is to use Roles.
So what does a role do? Essentially, it is a way of assigning a set of permissions to any entity that requires them. Using roles, you can delegate access to users, groups or AWS services when they need it.
Let’s look at a couple of examples, starting with a really simple one: giving a user the ability to switch to a different role.
Users Switching Roles
An IAM user has the ability to switch to another role - by doing so, that user will temporarily receive the permissions associated with that role. Note that the permissions associated with the role are not added to the user’s original permissions; instead, they replace the original permissions. Once the user has completed their tasks, they can switch back to their original permissions.
To create the policy, two main things are needed: a permissions policy and a trust policy. We’ve already seen what permissions policies do earlier in the post - they simply specify what permissions the entity will receive. I also mentioned trust policies briefly earlier on. A trust policy specifies which principal(s) can assume the role. So in our simple example, the trust policy would need to specify the user who actually wants to make use of that role.
In this example, I create a role called S3Access. Under the ‘Trust Relationships’ tab within this role, I will have a trust policy that is defined as follows:
Note that under ‘Principal’, I have put in the ARN (Amazon Resource Name) of my user. We covered ARNs in part 3 of this series.
Also note that where is shown, this would be the actual ID of your account (you didn't think I was going to show you my actual account ID, did you?). This policy states that the user 'araffe' within the account in question has the ability to assume the role.
OK, so if I first log in to the AWS console as ‘araffe’, I can see that I don’t have any access to S3:
But I know there is a role available that I have been given permission to assume, so I can switch to it using the drop down menu on the top right of the screen. I need to provide the account number where the role resides, as well as the name of the role.
Once I switch to the new role, I am able to see all of the S3 buckets in my account. I can also switch back to my original user when I am done and no longer need to permissions associated with the new role.
Assigning Roles to Services
As mentioned at the beginning of this section, roles can be assigned to services within AWS to allow access to other services. Let’s look at an example:
In this scenario, I have created a new role called S3AccessForEc2. This role has a permissions policy attached to it called AmazonS3FullAccess - you can guess what permissions this allows. There is also a trust policy within the role that states that the role can be assumed by EC2 instances (i.e. an EC2 instance can utilise this role and it will inherit the permissions specified in the policy).
So now, I can attach the IAM role to one of my instances (this can easily be done in the EC2 area of the console, or via the CLI / API). Once I’ve attached the role, the EC2 instance has full access to S3 inside my account.
So what’s actually going here behind the scenes? Well, when I attach the role to my EC2 instance, the instance receives a set of temporary credentials to enable it to access the resource (in this case, S3). You can actually view information about the role and temporary credentials from within the EC2 instance by querying the metadata. I’m going to cover metadata in a bit more detail in part 6, but for now let’s have a look at the role the instance has assumed:
You can see here that my EC2 instance is using the temporary credentials shown (access key ID, secret access key and token) to access S3.
Cross Account Access
Let’s say you have two AWS accounts - one for production and one for development. There are some resources in the development account (for simplicity let’s say it’s an S3 bucket) that you want users within the production account to use. How do you achieve this? One way to do it would be to just create a new user for each person that needs access within the development account and then give the username / password to those people. The trouble is, those people then have to log out of the production account and log back in to the development account in order to use the resources. It might work, but it’s not a particularly nice way of doing things.
Happily, there’s a way to grant access to resources between accounts using IAM, which means those users don’t need to continually log in and out of the different accounts. Here’s an example of how this works:
Let’s go through this step by step. The first thing I do here is create a new role inside the Production account (this account has the ID 33334444). The role has permissions assigned to it that allow full access to S3. The role also has a trust policy assigned that states ‘account ID 11112222 can assume this role’. What this actually does is creates a trust relationship between the two accounts, which means that rights can be granted to users in the development account (11112222) to assume the role.
The second step is to actually grant users in the development account the ability to assume the ‘S3AccessCrossAccount’ role in the production account. The easiest way to do this is by attaching a policy to the ‘DevUsers’ group (which in turn contains our users) that allows them to assume the role. That might look something like this:
Once this is in place, our development account users can switch to the ‘S3AccessCrossAccount’ role in exactly the same way I discussed in the previous section, which allows them to access our S3 resource.
Right, that’s enough IAM for now - like I said at the beginning, it’s important to understand IAM as so many other services rely upon it. Now that’s out of the way, we can move on and actually start to create some useful resources, starting with EC2 instances.
AWS provides a number of methods for interacting with the service, such as command line tools, CloudFormation (i.e. templated deployments - covered in a future post) as well as APIs / SDKs. However, it’s likely that for first time users and beginners, the AWS console will be the primary method for interacting with the platform.
AWS Console
The AWS console is a web based GUI that provides the ability to interact with the services available within AWS. For example, I could use the console to launch an EC2 instance (i.e. a virtual machine), view information about that instance such as the IP address that has been assigned to that instance and then terminate that instance once I am finished with it.
To begin, log on to the AWS console, after which you will be presented with a screen showing the AWS services available:
From here, you can search for the service that you are interested in, as well as see the recently used services (in my example, I have been using EC2, VPC and IAM). One other useful feature of the AWS console is the ability to pin commonly used services to the menu bar at the top of the screen. To do this, click on the pin button in the menu bar - you can see from the above screenshot that I have pinned EC2, S3, CloudFormation and EKS to my menu bar.
You can also set the console to the AWS region that you are working in. In my screenshot, you can see at the top right of the screen that I am working in the London (eu-west-2) region. Note that some AWS services (such as IAM) are not region specific, therefore you will see ‘Global’ in the top right of the menu bar when working with these services.
You can also view your account information using the console (under your account name in the top right menu option) - this link will take you to the AWS billing and management console which provides a whole host of information on billing, budgeting and payment options. If you wanted to grab your account number for example, you could also do this from the main account page.
Finally, the console provides a number of useful links to AWS documentation, tutorials and other helpful info - you can find this in the ‘Support’ link in the top right corner of the console.
AWS CLI
The AWS console is a great way to get started with AWS and get up to speed with the services available, but once you start to get more familiar with the platform, you’re likely to find the console a bit limiting in terms of speed and repeatability. One of the more advanced ways in which you can interact with AWS is using the AWS CLI.
The AWS CLI allows you to manage your AWS environment using a terminal rather than a graphical interface. This is not only quicker than clicking around a GUI, but it also means that you can perform a level of automation by scripting CLI commands. For example, you could create a script that contains all the commands necessary to create an EC2 instance, or create a new S3 bucket.
I’ll be covering the AWS CLI in a bit more depth in a later post, but for now let’s look at a couple of examples. All AWS CLI commands start with the aws keyword. You then generally specify the service you want to interact with, such as ec2, s3, rds and so on. Following that, you would normally enter the command corresponding to the action you want to take. So let’s say I want to list all the S3 buckets in my account (if you haven’t read the later posts covering these services yet, these terms might not mean very much to you now, but don’t worry too much at this point) - here’s the command I would use:
aws s3 ls
If I wanted to list out all the EC2 instances in my account I could use:
aws ec2 describe-instances
Let’s say I wanted to run a new EC2 instance - I could do that with the following command:
This command will run a single instance using a specific Amazon Machine Image (AMI), with an instance size of ‘t2.micro’ and using a key pair named ‘MainKey’. All of these terms will become clear later, but for now it’s enough to know that you can do pretty much anything in AWS using the CLI.
AWS SDKs
The real power of AWS lies in its programmability and ability to be automated. AWS provides a number of Software Development Kits (SDKs) that allow you to interact with the platform through code that you have developed yourself. SDKs are available for all the major programming languages, including Java, Python, Node.JS and Go.
Why might you want to use the SDKs? Typically you would use this approach if you want to integrate AWS services into the applications that you are writing - for example, your code could write information to a database service within AWS, such as RDS or DynamoDB.
CloudFormation
CloudFormation is a tool used to define infrastructure as code. The approaches we have discussed above all have their own pros and cons, but if you simply want to define the desired infrastructure within AWS, quickly and in a repeatable manner, CloudFormation is likely to be your best bet.
The idea behind CloudFormation is that you can model the AWS services you want to provision, such as EC2 instances, S3 buckets and so on, in a single script using a descriptive language. You can define this script locally and then when you are ready, upload it to AWS, at which point the platform will read the script and then provision the services and features that you have defined in the CloudFormatiom script.
One of the big advantages of using CloudFormation is that because your infrastructure is defined as code, you can treat your scripts the way you would treat any other code - checking it in to code repositories, using version control and so on. CloudFormation is also extremely flexible and powerful, allowing you to use many advanced features such as rolling back the ‘stack’ if there are issues, as well as the ability to detect ‘drift’ from the stack (i.e. if someone has come and manually changed something after the script has run).
OK, now that we’re familiar with the various ways in which you can interact with AWS, let’s move on and start looking at Identity and Access Management (IAM). See you in the next section!
So now you know what AWS is and you’re raring to get going. What do you need in order to start spinning up services? The very first thing you’ll need to do is set up an AWS account.
Accounts
In order to create resources within AWS, you must sign up for an account. An account contains all of your instances, virtual private clouds, storage and any other type of resource that you may wish to use. An account is completely dedicated to you (or other users you want to add) and you will be billed monthly for the usage within the account. It’s not possible for other customers and users of AWS to have any access into your account (unless you want them to of course).
Every AWS account has an account number associated with it - you’ll often need to use this account number for logging in, as well as giving other accounts access and other operations.
When you first create an account, you’ll need to provide a valid email address - this email address will become the username for the root account. I’ll be covering this in the IAM section so won’t go into any more detail here. You’ll also need to provide some information about payment methods and some other info such as phone number.
Once you have your account provisioned, you can dive in and begin creating AWS resources!
Billing
As mentioned in part 1 of this series, with AWS you pay only for the services that you use on a monthly basis - so if you have 10 EC2 instances (virtual machines) running, you are only paying for the time that those instances are in use. If you choose to shut those instances down, you won’t be charged for them (although you may still be paying for any underlying storage used by those instances).
It’s important to understand how the resources you are using are charged for - it may differ depending on the service you are using. For example, on-demand EC2 instances are billed based on the amount of time they are up and running for, with most charged on a per-second basis. If however you are using the Cloudfront content delivery service, pricing is based on a number of factors including data transferred out and the number of HTTP / HTTPS requests made. It’s fairly easy to estimate the costs you will incur for any given service using the AWS Simple Monthly Calculator, or the new calculator, which is in beta at the time of writing and therefore doesn’t yet have all services available.
Billing and Cost Management Dashboard
So how do users of AWS keep track of their spending in the cloud? Now that you have the ability to spin up virtual machines, storage, networking and a whole host of other resources within seconds or minutes, it would be very easy for spending to spiral out of control. Fortunately, AWS provides a number of tools that can help you to manage this and make sure that doesn’t happen. The billing and cost management dashboard is one of these tools.
The goal of the billing and cost management dashboard is to give you a quick ‘at-a-glance’ view of what you have spent on a month by month basis. It gives you a graphical breakdown of the services you have consumed, as well as a view of last month’s costs and forecasted costs for next month.
In the above example, the current month’s costs are just over $216 and you can see how these are broken down on the right hand side of the screen. You can see that this user has spent roughly $86 on EKS (Elastic Container Service for Kubernetes), with a similar amount being spent on EC2 instances. The rest of the month’s spend is between VPC costs, AWS config and a number of other services. Clicking on the ‘Bill Details’ button on the top right of screen takes you to a page where you can view the full bill for the month, as well as bills for previous months.
Cost Explorer
The billing and cost management dashboard is useful for a quick view of the month’s spend, however it doesn’t provide any detail about spending patterns or trends over time. The Cost Explorer tool is designed to fill that gap by providing reporting and analytics capabilities to help manage spending.
To get started, Cost Explorer has a few preconfigured views that allow you to look at the monthly costs per service, daily costs and a number of others. Here’s an example of a Cost Explorer view of the monthly costs per service:
The example above shows the monthly costs in my own AWS account, broken down by service (if you’re wondering why there are no costs shown before September, that’s because I didn’t start working at AWS until the end of August :-)). On the right hand side of the screen, I can filter this further based on service - so if I wanted to see the costs only for the London region (eu-west-2), I could apply a filter to show me that info. Once you have a view that you are happy with, you can save it for future use. You can also download the data from your reports into a CSV file.
Budgets
Being able to easily view spending information using Cost Explorer and the billing dashboard is great, but you are mostly (apart from the forecasting capabilities) viewing info about spending that has already taken place. What happens if someone spins up a large EC2 instance without your knowledge and you want to be notified if your costs exceed a certain amount? For this, you can use budgets.
A budget can work in a couple of different ways. Firstly, you can set up a cost budget that will monitor the costs within your account(s) against a threshold that you set and the notify you if that threshold is exceeded. In the example below, I am setting up a budget of $100 on a monthly basis, specifically looking at the EC2 service. If my EC2 spend exceeds $100, I’ll be notified.
The second option is to create a budget based on usage rather than cost. The following example shows how this works:
In the above screenshot, I’ve created a usage budget that measures the amount of outgoing data from EC2 - if my monthly budget of 5Gb of data is exceeded, I will receive a notification. Other examples of usage budgets are the amount of S3 storage used, region to region data transfer, or number of running hours for Elastic Load Balancers.
Organisations
Having one AWS account to manage is fairly easy - all resources are created in that one account and the billing is easy to manage through the tools mentioned in the sections above. However it’s quite common for customers (especially larger ones) to use multiple AWS accounts - sometimes many hundreds. The question is, how do you manage costs across a large number of accounts in this way? Do you just leave it to the individual teams to deal with their own bills? To help with this, the concept of Consolidated Billing and - later on - Organisations (and yes, strictly speaking I should be spelling it with a ‘z’ but I’m from the UK so I can’t help myself) were introduced.
The idea behind organisations is that you can take multiple AWS accounts and group them together to allow you to centrally manage them. Once the accounts are added to the organisation, you can do several things:
You can use consolidated billing to pay the bill for all accounts from a ‘master account’.
You can use policies to control what users of the accounts are able to do - for example, deny access to certain services.
Organisations have the concept of organisational units (OUs) to further group accounts together and make it easier to apply policies to those accounts. In the following screenshot, I’ve created an organisation called ‘Adam-Org1’, with two OUs underneath it (‘Adam-OU-1’ and ‘Adam-OU-2’):
I also have two accounts - I’ve added one into OU-1 and the other into OU-2 (only OU-1 shown here):
The process for adding accounts into an organisation is straightforward - you can either create new accounts from within the organisation view, or you can invite existing accounts to join the organisation, after which an administrator from the invited account needs to approve.
Service Control Policies
Service Control Policies (SCPs) allow administrators within an organisation to control exactly what users are allowed to do within an individual account. Technically, an SCP determines what administrators can allow the users or roles in their account to do - so if an SCP allows the ‘AttachLoadBalancers’ action for the EC2 Auto-scaling service, then the administrator of the account to which the SCP is attached can grant permissions for that action to users and roles within the account.
Here’s a quick example: in the screenshot below, I have created a ‘deny’ SCP which includes the ‘CreateSubnet’ action for the EC2 service. If I attach this SCP to an account (or to an OU that includes that account), the administrator will not be able to grant permissions to any role or user for that action, therefore no-one within that account will have the ability to create subnets.
Amazon Resource Names (ARNs)
To finish off this post, I’ll quickly touch upon Amazon Resource Names (ARNs). An ARN is a unique identifier given to all resources that you create within AWS. It’s a very common requirement to need to be able to refer to other resources (for example, when you need to refer to a specific IAM role from another service) and the ARN is often used for this purpose. Let’s have a look at a typical ARN and break it down into the various parts:
The exact format of the ARN varies depending on which type of resource you are talking about. As an example, the S3 (Simple Storage Service) doesn’t require an account number or region in the ARN, so you would typically see something like this, where those values are omitted:
arn:aws:s3:::my_bucket
OK, that’s it for this post - hopefully you found this useful. Next up, I’ll be introducing you to the different ways you can use to interact with AWS, such as the console, CLI and SDKs.
