AWS multi-account setups reloaded

Handling multiple AWS accounts turns out to be a challenge, especially for developers who are supposed to work in such environment as part of their everyday work routine.

Handling multiple AWS accounts turns out to be a challenge, especially for developers who are supposed to work in such environment as part of their everyday work routine. For AWS operators it might be daily business to switch between e.g. dev, staging and production accounts or just deploy things seamlessly between them. However, from my experience a lot of people are struggling by using aws-cli or other aws related tooling if it comes on multi account structures in combination with an unified IAM user base.

This blog post should give a simple high-level overview on the general idea of multi AWS accounts. It doesn’t deal with details as much as possible in order to shed some light on the overall concept. Moreover it is an aftermath of the DevOpsCon conference earlier this year. If you are interested on my slide deck you can download it here.

Let’s have a look on the internals and give some recommandations. Lets start with the foundation and explain how it works in the most easy fashion:

Imagine you have three AWS accounts:

  1. User Account - here is where your teammates have an IAM user. They can log in via AWS console, create there own API access keys, setting MFA and changing their password. That’s basically it.
  2. Testing Account - most of the developers can do more or less everything here.
  3. Production Account - only senior engenieers are allowed to change things here. Everybody else can only sit and watch or doesn’t get access at all.

Organisational unit

A developer, who is about to do any kind of operation inside the Testing or Production account, needs to get access to it. The problem here is that his personalized IAM user only exists inside the User account. To do anything somewhere else, he/she needs to do one thing: getting (temporary) credentials to access resources inside Testing or Production. This is done by assuming an IAM role inside Testing or Production by doing an STS:assume-role API call. Technically two things are necessary in order to do this (source - Brian Wagner, AWS Security Blog).

DevAccount

  1. The IAM user needs to have the permission to assume the external role from within the User account.
  2. The IAM role inside the external account needs to have a policy that grants a user of the User account the permission to assume the role.

In the end, both accounts (User and Testing/Production) must allow the cross account access. This is done by defining IAM policies and attach it to users on the one end and roles on the other. Resources in AWS have the account ID as part of their Amazon Resource Name in order to identify each other e.g. arn:aws:iam::123456789012:user/jan.nabbefeld@kreuzwerker.de.

IAM policy to assume a cross account role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "arn:aws:iam::<TARGET ACCOUNT ID>:role/admin-cross-account"
     }
  ]
}

IAM trust relationship of a cross account role

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<IAM USER ACCOUNT ID>:root"
      },
      "
      Action": "sts:AssumeRole",
    }
  ]
}

So how do you get such temporary credentials? There are multiple ways to do this which depend on things you going to achieve. AWS SDKs using credentials in a specific order of appearance. Most straightforward method is setting environment variables. Alternatively you can work with profiles as part of your ~/.aws/credentials file.

So as an AWS operator or DevOps what should you do in order to keep everything straight?

  1. Usually AWS account structures should stay straight forward. Don’t overdo and create thousands of AWS accounts just because you can. Keep in mind that you should set a root password for each account and set MFA active. Currently this is something that can’t be done in an automated fashion with ease.
  2. A reduced set of roles per account is good enough for most scenarios. That keeps complexity low and feels like a sane approach. IMHO three kinds of user roles do the trick: administrator, operator and spectator permission sets. Which permissions make sense for which role is in the end up to you. Given the fact that administrators are allowed to do everything, operators are flexible. I would make sure to disallow expensive operations like buying reserved instance, modifying your AWS organization and configuring Cloud HSM. Finally spectators should have a minimal set of permissions. Ideally they are bound to read only operations.
  3. Use MFA also on API access level. That indeed drives complexity but increases security a lot. There are tools like awsu that help you to implement a secure MFA API access.

You might think AWS IAM tends to be complex and complicated? Yes, you are totally right, it’s confusing, limited in some (unexpected) extends and hard to test. Nevertheless, it is one of the AWS cornerstones that deserves a lot of attention.

Credits for cover image go to: Fargo (Tv-series).

Author

Jan Nabbefeld