In our company, we have a two-steps workflow to plan-then-apply the changes.
The first step generates a TFPlan (tf plan -out tfplan). It uses a read-only role on the infra, as well as a read-only role on the backend (RO on S3 + RW on DynamoDB).
The second step takes this TFPlan as an input, and tries to apply it using a read-write role (tf apply tfplan) for both the infra and the backend (to push the new state).
This is where our issue happens, it seems that the TFPlan stores the changes as well as the backend configuration (including the role to assume), thus causing an unauthorized error because it uses the read-only role to push the new state.
Is it possible in any way to force Terraform to re-read the backend configuration before applying from a TFPlan?
Or modify the TFPlan to use the read-write role?
It sounds like you’ve set the assume role configuration directly inside your backend configuration, and so Terraform has copied it into the plan file because that’s how Terraform remembers what backend configuration the plan was created with.
If you want to change this setting between plan and apply then you will need to set it “out of band”, such as using environment variables instead of directly setting it in the configuration. Environment variables are separate from the backend configuration, so you can change them however you like between plan and apply as long as you make sure that the new values are equivalent enough to the old values for the apply operation to make sense.
For AWS in particular, I think the expectation is to specify the role to assume in the AWS configuration file and then either place that file in the default location or set an environment variable to specify the location of the file.
Terraform providers and state storage backends aim to follow the conventions of the platform they are integrating with, and so the AWS provider and S3 backend aim to get configuration and authentication information in the same way that AWS CLI does. I would suggest configuring AWS CLI to assume the role you want to assume and then the relevant Terraform components should follow and use the same strategy.
(For AWS in particular the Terraform features achieve this by using the official AWS SDK, which is also used by AWS CLI.)
So, by your sayings, I guess my solution is not to use the assume_role.role_arn of the S3 backend configuration, but rather setup my AWS credentials before running any Terraform command?
But I can’t do that as my Terraform code and my backend use 2 different roles, and there are some data sources in the code using even another role.
So there is no possibility to use an environment variable for the role to use?
Then if I can’t modify this, isn’t it possible to modify the tfplan file directly to use some role instead of another?
The AWS configuration format has the concept of “profiles” to allow you to configure and use multiple sets of credentials and other settings at the same time. You can select a profile to use by name, using an argument in the provider or backend configuration.
Anything that is in the backend configuration must remain fixed between plan and apply. The design intention is that you use the backend configuration only for the fixed information about where the state is stored, and use separate configuration like the AWS CLI configuration for anything that varies between runs.
The S3 backend documentation includes a section Multi-account AWS Architecture which describes one possible way to set things up to use different credentials for the backend vs. the provider across different environments, which is a little different to what I’ve suggested so far. I don’t think what it describes is exactly right for the situation you are considering, but perhaps that section will give you some other ideas to consider.
Thanks for your answer.
Sadly we can’t use AWS profiles to achieve this, as we assume multiple roles during the plan.
I’ve read the article you’ve linked, thanks for that, but it doesn’t help us that much.
We have already these accounts/roles/links/limitations/ACL/etc set up.
The real issue is related to the tfplan file that we can’t modify on-the-fly.
I’ve tried using variables to select which role to use in the backend, but it’s not permitted by Terraform to use dynamically configured backends (by using vars).
For now we can’t use read-only roles because of that heh.
Don’t you think that it could be possible to edit the tfplan directly to modify the roles it uses?
I coudn’t find any documentation on the internet about this file format, and I feel like the tf show -json <tfplan>'s output isn’t comprehensive at all.
As a final comment to close this unsolved issue, do you know if there are any resources or documentations about the tfplan file format?
We discussed and came to the conclusion that it is ok for us to maintain a script to edit this file with our use-case, even if we have to fix it every couple months with the new TF versions.
Even better, I don’t think we are the only ones in this situation. Do you know how it is done by the others in any way?