Error refreshing state: AccessDenied: Access Denied - Multiple AWS Organisations

Terraform
1

I have inherited a terraform project, version v1.2.4 to build infrastructure in AWS and having issues running the terraform init -backend-config=backend.auto.tfvars command so I can then run things like terraform plan and terraform apply.

The way we have it structured is:

  • AWS Master account (root organisation)
  • AWS production child org
  • AWS staging child org

I have created an IAM user with Administrator access via the AWS Master account, which has a trusted policy to access both staging and production.

Then locally I have configured was cli and the ~/.aws/credentials file like the following:

[default]
aws_access_key_id=<fill creds from master account ()>
aws_secret_access_key=<fill>

[xxxxxx-staging]
role_arn = arn:aws:iam::99999999999:role/admin
source_profile = default
region = us-east-1

[xxxxxx-prod]
role_arn = arn:aws:iam::99999999999:role/admin
source_profile = default
region = us-east-1

Then I will log in via console to the IAM user (master account) assume the role of staging for instance, so then my IAM user has full admin access to make changes to the Staging child org.

Once this is done (and I have confirmed it’s all working correctly with some AWS commands) I have full access via AWS cli to staging. (tested using aws sts get-caller-identity and even aws s3 ls.

I will export the profile I want to use like export AWS_PROFILE=xxxxxxx-staging

Everything works as expected, I can download terraform secrets stored in S3 etc but as soon as we run terraform init -backend-config=backend.auto.tfvars

It will fail pulling the remote state from s3 it will only ever use the master IAM details the aws_access_key_id and aws_secret_access_key and not the assumed role.

I am at a loss on why this happens and it seems it’s something within terraform and not anything code wise.

The errors I am getting are:
Error refreshing state: AccessDenied: Access Denied
status code: 403, request id: xxxxxxxxx, host id: xxxxxxxxxx

Where I am getting forbidden from s3/GetObject

I even before running the init command ran the aws sts get-caller-identity command first and that is using the correct details/role but as soon as the init command runs it reverts to not using it.

Driving myself mad with this one!

Other notes: We are using docker also so this is running within docker and not my local machine. But docker has got access to the correct .aws credentials file