Supporting Local and Remote Terraform development

Hi all!

When developing a new Terraform code, what are some of the best practice to allow both local and remote development with S3 backend?

locals {
  role_arns = {
    dev     = var.iam_role_to_assume_dev
    test    = var.iam_role_to_assume_test
    staging = var.iam_role_to_assume_staging
    prod    = var.iam_role_to_assume_prod
  }
}

workspace = merge(lookup(local.env, terraform.workspace))

provider "aws" {
  region = "ap-southeast-1"

  assume_role {
    role_arn = local.role_arns[terraform.workspace]
  }
}

terraform {
  backend "s3" {
    bucket         = "terraform-backend-tfstate"
    key            = "terraform-route53.tfstate"
    region         = "ap-southeast-2"
    dynamodb_table = "terraform-state-lock"
  }
}

Would it be fair to assume that the local.role_arns[terraform.workspace] role should have enough permissions to allow access to the TF backend as well as to provision resources?

Also, I assume that if I would want to run TF locally, I would have to export the iam_role_to_assume_dev role in my shell? Like export TF_VAR_iam_role_to_assume_dev=""?

Any suggestions are more than welcome.

Thanks!

This bit doesn’t look like valid Terraform syntax to me?


There are multiple things local development can mean. One of them, is to just experiment with configurations without applying, using only local state. To support this use-case I find it helpful to always put my terraform { backend "..." { ... } } block in a separate backend.tf file, so that I can simply delete it to switch to local backend, and retrieve it from Git to undo.


Even if you have both an S3 backend and an AWS provider, these are very separate pieces of Terraform functionality - with entirely separate configuration.

It is entirely up to you whether you use the same AWS credentials for both - or totally different ones. Whatever works better for your situation.

You can have a dedicated IAM role for accessing the backend s3 bucket. If your remote flow and local users can both assume the role then you won’t have any problems

For example:

terraform {
  backend "s3" {
    bucket         = "terraform-backend-tfstate"
    key            = "terraform-route53.tfstate"
    region         = "ap-southeast-2"
    dynamodb_table = "terraform-state-lock"
    role_arn       = "arn:aws:iam::123456789012:role/TerraformS3BackendRole"
  }
}

Adding in the environment variable is an elegant solution. The other way is to create a file ending in .auto.tfvars with the local variables. If you use a standard name like localrole.auto.tfvars you can add it to .gitignore and then it won’t get commited.

1 Like

Thanks, @cregkly !

Using the assume role option is the best one I believe.

However, I’m having a bit of trouble now with my CI/CD (Github Actions) pipeline.

I have the DynamoDB table for the TF lock and the S3 bucket for the TF state file in a single account (Account A).

  • AccountA = S3 Bucket and DynamoDB Table
  • AccountB = Role that Github Actions is assuming

Issue:
The AccountB role does not have access to the DynamoDB table in AccountA.

I think the only solution here is to deploy a DynamoDB table for each account.

Just add a policy to allow access to the dynammodb table as well.

The DynamoDB table is in a different account than the one Github is assuming. It doesn’t work.

I use a bucket and dynamodb table in one account and use it from all my other accounts in terraform.

As long as the role you are running under can assume the backend role it will work.

Hmm… Let me try to better explain what’s happening.


Workflow is something like this:

From Github: (not working)

  • terraform init: GithubActions → OIDC Connection to AWS → github-oidc role in each account → terraform-backend-role (PROD account) > DynamoDB|S3
  • terraform plan: GithubActions → OIDC Connection to AWS → github-oidc role in each account → DynamoDB|S3 ← It doesn’t know it needs to assume the terraform-backend-role role in order for it to get access to the DynamoDB table and S3 bucket.

From local: (working)

  • terraform init: Local laptop → aws sso login credentials → terraform-assume-role role in each account → terraform-backend-role (PROD account) > DynamoDB|S3
  • terraform plan: Local laptop → aws sso login credentials → terraform-assume-role role in each account → terraform-backend-role (PROD account) > DynamoDB|S3

prod account:

  • DynamoDB Table for TF locks
  • S3 Bucket for TF state files
  • github-oidc role for Github Actions
  • terraform-assume-role for local TF development (When running TF locally you assume that role)
  • terraform-backend-role when running terraform init you assume this role

test account:

  • github-oidc role for Github Actions
  • terraform-assume-role for local TF development (When running TF locally you assume that role)

dev account:

  • github-oidc role for Github Actions
  • terraform-assume-role for local TF development (When running TF locally you assume that role)

Role github-oidc:

  • AdministratorAccess AWS managed policy
  • Custom policy to allow it assume the terraform-backend-role role
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/terraform-backend-role"
            ],
            "Effect": "Allow"
        }
    ]
}
  • Trust policy to allow Github Actions assume this role via OIDC connection
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<AWS ACCOUNT-ID:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                }
            }
        }
    ]
}

terraform-assume-role:

  • AdministratorAccess AWS managed policy
  • Custom policy to allow it assume the terraform-backend-role role
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": [
                "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/terraform-backend-role"
            ],
            "Effect": "Allow"
        }
    ]
}

Trust relationship allows SSO users to assume this role. This is used when running terraform init or terraform plan locally:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<AWS ACCOUNT-ID>:saml-provider/*"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:AssumeRoleWithSAML"
            ]
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<AWS ACCOUNT-ID>:root"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

terraform-backend-role:

  • custom policy to allow access to the DynamoDB table:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:DeleteItem",
                "dynamodb:PutItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:ap-southeast-2:<AWS ACCOUNT-ID>:table/terraform-state-lock",
                "arn:aws:dynamodb:ap-southeast-2:<AWS ACCOUNT-ID>:table/terraform-state-lock/*"
            ],
            "Effect": "Allow"
        }
    ]
}
  • custom policy to allow access to the S3 bucket:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::foo-terraform-backend-tfstate",
                "arn:aws:s3:::foo-terraform-backend-tfstate/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Trust policy allows the terraform-assume-role and github-actions-oidc-role to assume it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/terraform-assume-role",
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:role/terraform-assume-role",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:role/terraform-assume-role"
                ]
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/github-actions-oidc-role",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:role/github-actions-oidc-role",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:role/github-actions-oidc-role"
                ]
            },
            "Action": [
                "sts:AssumeRole",
                "sts:AssumeRoleWithWebIdentity"
            ]
        }
    ]
}

Just create one terraform-backend-role in the account with the S3 bucket and dynamodb table. Add in the trust to the other accounts and your runner roles.

In your terraform backend you specify this role in this account to assume.

Don’t create a terraform-backend-role in each account.

Yep, the terraform-backend-role only exists in the Production account, which is the same account as the S3 bucket and DynamoDB table.

terraform-backend-role:

  • custom policy to allow access to the DynamoDB table:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "dynamodb:GetItem",
                "dynamodb:DeleteItem",
                "dynamodb:PutItem"
            ],
            "Resource": [
                "arn:aws:dynamodb:ap-southeast-2:<AWS ACCOUNT-ID>:table/terraform-state-lock",
                "arn:aws:dynamodb:ap-southeast-2:<AWS ACCOUNT-ID>:table/terraform-state-lock/*"
            ],
            "Effect": "Allow"
        }
    ]
}
  • custom policy to allow access to the S3 bucket:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::foo-terraform-backend-tfstate",
                "arn:aws:s3:::foo-terraform-backend-tfstate/*"
            ],
            "Effect": "Allow"
        }
    ]
}

Trust policy allows the terraform-assume-role and github-actions-oidc-role to assume it:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/terraform-assume-role",
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:role/terraform-assume-role",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:root",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:role/terraform-assume-role"
                ]
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/github-actions-oidc-role",
                    "arn:aws:iam::<TEST AWS ACCOUNT-ID>:role/github-actions-oidc-role",
                    "arn:aws:iam::<DEV AWS ACCOUNT-ID>:role/github-actions-oidc-role"
                ]
            },
            "Action": [
                "sts:AssumeRole",
                "sts:AssumeRoleWithWebIdentity"
            ]
        }
    ]
}

My mistake, It was hard parsing your code and I think I mixed your terraform assume role with the backend one.

Did you update all of your backend terraform blocks to use the role?

role_arn = "arn:aws:iam::<PROD AWS ACCOUNT-ID>:role/terraform-backend-role"

I pretty much have the same setup with github runners and assume role polices.

All good!

Yes, all the backend configuration are using the backend role…

Do you use a different role for the provider itself? Because I do…

Yeah, the backend role is only used by terraform.

I assume a role into the account when I run locally, however our git hub runner is in account.