Hi,
I’m wondering whether it’s possible to tell data tf_remote_state to use/inherit the backend configuration of the current state? Or, to access/read the backend configuration within Terraform code in order to explicitly propagate some of it into data tf_remote_state.config
Any feedback on this appreciated!
Hi @lucendio,
There is currently no mechanism for doing this. The backend configuration included in terraform_remote_state is assumed to be totally unrelated to the one associated with the current configuration.
If your intent is to reduce repetition of the settings in question, you might be interested in the pattern of using a Data-only Module to encapsulate retrieving the data from remote state. That way the settings for getting that data are defined only inside the module, and you’ll be free to change them in a future version of the module. This also gives the option of changing to no longer use terraform_remote_state at all and to get the data some other way in a later version.
Thanks for the help and your suggestion.
Would you mind elaborating on this and show-case how you would envision this? I cant really see how terraform_remote_state in a different module can know about some backend config other then by providing this through input variables. But then, it would’nt make a difference whether terraform_remote_state is in a module or not.
Are you familiar with the TF code base to an extend where you can assess whether this is a simple change or a more complex one?
Hi @lucendio,
The intent of my suggestion is that you have the backend configuration hard-coded in the child module, but because it’s a separate module you can then include it from many other configurations so that it’s still hard-coded only in one place.
Okay, then I actually did no misunderstand you. Unfortunately this would not be an option in our case. We use a client-side provided encryption key for the state file, which means we’re looking for sth that only live during runtime or does not leave the system. And, I would like to avoid adding that encryption key to the state file. I guess, then the only way is to provide backend config not just with -backend-config init parameter but redundantly a second time as input variable.
Would it be worth opening a PR/Issue about this?
Usually backends are designed to receive credential-like materials out-of-band via mechanisms like environment variables, credentials files, or helper programs that are conventional for whatever system the backend is interacting with. For example, the “s3” backend aims to look for credentials in all of the same locations that the AWS CLI would look, so that users can obtain their credentials in whatever way makes sense for the specific situation and make them available to all AWS-related software at once.
You didn’t mention which specific backend you are using here, but it sounds like this encryption key belongs to this same category of things I’d expect to provide to the backend out-of-band. Does the backend you are using have a mechanism for doing that? If not, I think addressing that limitation in the backend is the most likely way this would be resolved, to make this backend consistent with the usual model for credential materials.
Hey @apparentlymart,
it’s s3 and the sse_customer_key I implied. And, you were right, the custom key can indeed be provided out-of-band. To be honest, I just spotted this, when looking up the link for this message. I’m sorry!
In any case, maybe you will have the opportunity to bring up the topic of backend inheritance for tf_remote_state internally, if it makes sense to you.
A last question on this matter. If the key is not explicitly provided via backend-config, I guess, AWS_SSE_CUSTOMER_KEY has to be set not only for terraform init but for apply as well, right?
Yes, the credentials to access the backend must be available for any operation that reads or writes state snapshots, which includes terraform plan and terraform apply.
I suppose if you were previously setting that inline in the backend configuration it would’ve appeared that you only needed to set it during terraform init, because the init command was saving that setting for you in .terraform/terraform.tfstate where the active backend configuration is cached.
When providing credentials out-of-band Terraform doesn’t duplicate them into that cache file because we assume they are probably sensitive and thus it would be undesirable to write a copy of them out to disk silently, and so you will indeed need to preserve the out-of-band setting between the different operations yourself somehow.