How to manage devs permissions in AWS across clients and products

Does the phrase “Hey, I don’t have iam:passRole permission” sounds familiar ?

Managing permissions is always a tedious task for any new project. You don't want to give too many permissions nor do you want to become a bottleneck, inundated with dev requests to increase privileges because they can’t work efficiently.

From these conflicting forces, we tried to define a workflow which would respect security concerns and dev flexibilty

Statement

We have a single master account for all the company. This account has mainly 2 groups "dev" and "admin". The dev group has the PowerUserAccess policy, it means they can do everything but manage IAM. We use the trusted advisor service to monitor costs and check that nobody is wasting resources. We also have a user "ci" that has only the permissions necessary to deploy.

While everything was working almost as expected, ultimately we got stuck because we didn’t want to give developers the roles necessary to deploy in the environment we had established (staging or prod). Ideally, only the ci and admins would have access to those environments. But how then could a dev test what they’d written in a real AWS environment? When you deploy a project, you need to create a specific user and devs did not have the role that would allow them to do it (nor should they). To improve this we defined what the ideal system would look like for us.

• Every dev must be able to work without having to constantly modify their permissions
• Staging and production should only be accessible by the CI and admins
• Clients must be isolated
• Products must be isolated
• Dev, staging, test, prod environment must be isolated

Organizations and switch-role to the rescue

To achieve this, we decided to create new organizations for each client, each product and each developer under our master account. An organization is a brand new account managed by another account. Each account/organization has its own quotas, users, roles, resources etc… To easily access these sub accounts, we need to use the switch role feature. Basically it allows a user from the master account to assume a role in the sub account.

Here is a schema defining a user who can access to 2 differents organizations each one containing a product environment.

AWS permissions accross accounts

In the schema above, we see that a user can switch to 2 roles with different permissions. In the staging environment, he has an admin role. This allows him to have full access in the account. On the other hand, for the production environment he only has the observer role, giving him read-only acces to some specific resources. It's important to note that the user doesn't exist in sub accounts, he only assumes a role.

The master account is the trusted account and sub account are trusting accounts. This means that sub accounts only allow users from the trusted account to connect.

Each organization has its own account id and if you want to easily switch between accounts you need to add the role to your top navbar. The switch role link will bring you to this configuration page. Simply add the account id, the role you want to assume and the display name which will be shown for your available roles. You can also build a preconfigured link to share with members of an environment like this one : https://signin.aws.amazon.com/switchrole?account=1234567890&roleName=observer&displayName=production

AWS switch role configuration

Configure your profile

Once you have several profiles it would be cool to be able to use them from the cli. Aws recommends adding the following configuration in your `~/.aws/config` to be able to use this new profile with aws cli. You will be able to use `aws — profile staging` to execute a command in your staging org.

If you need to use those credentials in other tools that don't rely on aws cli like we do with serverless framework, we figured we need to add them to `~/.aws/credentials`

[profile default]
region = us-east-1

[staging]
role_arn = arn:aws:iam::<staging_account_id>:role/admin
source_profile = default

[production]
role_arn = arn:aws:iam::<production_account_id>:role/observer
source_profile = default

Conclusion

Now every developer is happy with his own environment and has no permission issues while he works in his account.

We now have a much better isolation for each environment even within the same project. It's easy to see how much each service and user costs. Splitting dev accounts will make them fall under the free tier account so it won't increase the company expenses. It's also easy to allow groups from the master account to access sub accounts.

The drawback is that you have to maintain many separate environments. The CI/CD deploys and runs migrations for databases but in your personal account you have to manually perform such operations unless you move them from the ci config to the project config.

Also you have to be very careful with permissions and I recommend that you explicitely set assumeRole on each role that's needed. If a user can assumeRole on every resource, which we did by giving the PowerUser policy, then he could easily guess the role name "admin" for the production environment. Therefore it’s essential that there is no blanket granting of the assume role permission in the trusted org, but rather specified at the user or group level since this is the only way to secure access to the different orgs from the master org.

Finally, if you subscribe to AWS support, it won't work across accounts.