Let me ask you a difficult question: how often are you really rotating your IAM credentials? Secondly, if there were a security breach and someone had access to those IAM credentials, how quickly would you be able to detect it?

Security is something that nobody in tech likes to take for granted and is an area that is constantly changing and evolving as more and more threats are detected every day.

About 8 months or so ago I was thinking to myself “I really need to update and rotate the IAM Credentials I use for this blog”. I was namely thinking of this because while it is good security practice to rotate your keys, unfortunately, there’s no “automatic” way to do this within AWS, short of writing a Lambda and hooking this into Secrets Manager and/or SES for reminders of key rotation. I was researching the most straightforward way to automate these key renewals say every 90 days and then auto-uploading those keys to the services that use these particular keys (in my case Terraform Cloud and CircleCI). However, after spending a few hours digging in, I quickly realized that this would not be that straightforward and that it was actually way faster for me to simply log into my AWS account every 90 days, rotate the keys, and then upload these to CircleCI and Terraform.

However, I recently listened to this AWS Bites podcast episode and thought “hrmm…ok, maybe I’ll just switch up my blog’s CI/CD to GitHub Actions so I can use this awesome OpenID Connect (OIDC) functionality”.

And then I read this great CircleCI blog post 😄

Understanding why OIDC is superior to IAM credentials

So the standard process has typically been to: create an IAM user with programmatic keys, give that user permissions of least privilege, and then upload the IAM credential keys to the service provider that will be making changes to your AWS infrastructure (again in my case Terraform Cloud and CircleCI) on your behalf.

When you really think about it, storing your IAM credentials in any service provider has its limitations. I like to think of providing IAM credentials to a service similar to giving my credit card details to a service (e.g. CircleCI). I trust CircleCI to not do anything nefarious with my details, but if my card details are stored in the service and that service for whatever reason becomes compromised, that credit card can be used by anyone (or any service) that has the details. This is no different from IAM credentials and provides us with a good reason as to why you always want to pair down the permissions.

side note: please, please don’t create an IAM user with programmatic keys that has AdministratorAccess permissions to your AWS Account that you then upload those IAM credentials to these third-party services. You are basically giving anyone or any service the ability to create anything (well…within service quota limits) within your AWS account by doing so. You personally don’t want that or want to deal with an unexpected large AWS bill shortly after! 😅

Continuing the credit card analogy; so instead of giving our credit card details to a service, what if there is a way to give only card authorization to that one service? Well, that sounds more like OIDC! The benefit is that you are only authorizing/allowing the service to use your credit card, meaning that there are no credit card details to be leaked, only card authorization details. If the card authorization details were somehow leaked, these are actually kind of worthless, as only that particular service is authorized to use them, not some other random third party! 🎉

I’m confident there will come a day soon when some hacker will figure out how to spoof the service that has the OIDC Identity provider authorization permission (e.g. a hacker spoofing CircleCI can then access your AWS infrastructure based on the policies attached to the OIDC provider), but for now this setup seems a whole lot more secure than storing your IAM credentials directly at the service and having to remember to rotate your access keys every so often. Further, if the IAM credentials are leaked, you really have to rotate them quickly or risk your AWS account/services being compromised.

How to setup an OIDC Identity Provider with CircleCI in AWS

  1. Start by logging into CircleCI and then go to Organization Settings > then copy your organization ID
  2. Log into the AWS console, and head to IAM.
  3. Go to Identity Providers (under Access Management) and click “Add Provider”
  4. Choose “OpenID Connect”
  5. In Provider URL enter https://oidc.circleci.com/org/<INSERT ORGANIZATION ID HERE> (e.g. https://oidc.circleci.com/org/th1s-i5-n0t-a-0rg1d) and then click “Get Thumbprint”
  6. In Audience, enter the same Organization ID
  7. Click “Add Provider” and the provider should be created!
  8. Next, go to Roles (under Access Management)
  9. Click “Create Role” and select “Web Identity”
  10. In Identity Provider select the CircleCI identity provider that was just created.
  11. In Audience, select the Organization ID and click Next to add permissions
  12. In Permissions, you will be selecting or creating policies based on what your CircleCI pipelines should have permission to do. This will vary case by case. For example, my CICD pipelines need only access to S3 and CloudFront services to publish my blog posts.
  13. Click Create Role and when complete, copy this Role’s ARN, you are good from the AWS perspective!

Using OIDC as your AWS credentials in CircleCI

Before making any changes to your CircleCI pipeline, as per the OpenID Connect Tokens doc, you will need to create a context for your job, which can be done using these steps.

Once you have a context created and added to the workflow section of your pipeline job, you can now edit your CircleCI pipeline to include the OpenID Connect token to utilize our newly created IAM role. For example, using these directions, my .circleci/config.yml file was changed from this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: Push to S3 bucket
          command: cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
      - run:
          name: Invalidate Cloudfront
          command: aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

to this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    environment:
      AWS_DEFAULT_REGION: <ENTER YOUR REGION HERE>
      AWS_ROLE_ARN: <ENTER YOUR IAM ROLE ARN HERE>
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: authenticate-and-interact, Push to S3 bucket, Invalidate CloudFront
          command: |
            # use the OpenID Connect token to obtain AWS credentials
            read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN \<<< \
              $(aws sts assume-role-with-web-identity \
               --role-arn ${AWS_ROLE_ARN} \
               --role-session-name "CircleCI-${CIRCLE_WORKFLOW_ID}-${CIRCLE_JOB}" \
               --web-identity-token $CIRCLE_OIDC_TOKEN \
               --duration-seconds 3600 \
               --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' \
               --output text)
            export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
            # interact with AWS
            aws sts get-caller-identity --no-cli-pager
            cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
            aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

So for the eagle-eyed reader, you’ll notice a few interesting pieces. Firstly, the addition of --no-cli-pager is necessary probably only for the docker image used paulmarsicloud/hexo-aws:2.1. This is a custom docker image I created that has both hexo and the aws-cli installed, as this blog/site uses the Hexo platform. By adding the --no-cli-pager option, I no longer receive a Unable to redirect output to pager. error 👍

Secondly, I had to combine the Push to S3 bucket and Invalidate Cloudfront steps into one during testing, as it seems that the read and export commands need to be run per step. I tried having just a singular authenticate-and-interact step, but subsequent steps that ran aws cli commands would fail with a Unable to locate credentials. You can configure credentials by running "aws configure" error. I will need to sit down and spend some further time testing out and potentially refactoring my CircleCI steps and pipeline as I am sure there is a way to somehow cache or pass the credentials from one step to another, rather than having to write and execute those read and export commands each time you want to run an aws-cli command. When I have the answer, ya’ll will be the first to know!

In a way, it makes more sense to have these S3 and CloudFront AWS commands run in one step, even though for readability purposes, I personally like to have a “one command per step” layout for my CI/CD pipelines. It’s truly a “6 in one, half a dozen in the other” kinda deal though.

Here’s a link to the gist of my final updated .circleci/config.yml if you need it.

Terraform Cloud

Unfortunately as per this discussion Terraform Cloud does not support OIDC Identity. This is discouraging to see and as a result, I will be moving my Terraform state out of Terraform Cloud shortly and will set up an S3 Bucket/DynamoDB table for the state/lock. I initially set up this blog with Terraform Cloud to try it out and have automated Terraform runs, but now I will be switching to CircleCI (or GitHub Actions) so that I can utilize the awesomeness of OIDC!

Benefits of OIDC over IAM credentials

  • Sounder sleep knowing that my IAM credentials won’t be leaked for someone or some service to use and compromise my AWS infrastructure
  • No need to set up anything custom or any reminders to manually rotate the keys. All authorization is now done via the OIDC Identity Provider and corresponding IAM role - it was a rad feeling deleting the IAM credentials I had stored in CircleCI and still having the pipeline runs work!
  • No way of accidentally forgetting to not store your IAM keys in version/source control (this happens to folks all the time, even though everyone warns about it)
  • OIDC providers are verified by AWS, so there is a super low risk that an OIDC Providers “Thumbprint” would be spoofed for a particular large service (like CircleCI or GitHub Actions). When your IAM credentials are created and are then used for any service, anywhere, AWS doesn’t ever verify who exactly is using them. There is an implicit trust that if a user or service has the IAM credentials, they are authorized to use them. It is a totally different ball game with OIDC!

Recap

  • Think of IAM Credentials as your Credit Card vs an OIDC Provider as authorization to use your Credit Card 🙂
  • OIDC Providers are quick and simple to setup in AWS and roll into your new or existing CircleCI pipelines
  • You never need to rotate OIDC providers…you can’t say the same for IAM Credentials!