Hi all,
TL;DR; need a method to rapidly change a resource property value (stored and deployed) that is controlled by TF but have no TF drift and also allow the state to be redeployed from scratch with the same values as before and not default back when those values have been changed.
I am trying to find the best way to allow variables values to be dynamically changed without creating Terraform state drift for the AWS environment we are in.
The main overarching need for this is to allow a SaaS product to have\offer a quick way for technical staff\clients to modify their existing product (even maybe to create one) whilst still allowing IaC pipelines to run from CICD (Gitlab in our case) without issue but also not cause state drift, i.e. Terraform trying to undo a change that was made outside of its own process.
I have many use cases for this ranging from full creation of a product down to the simplest one which is GP3 IO value.
Sticking with the simple use case, a technical staff member notices the IO load is high on an instance backed by a GP3 volume. In the current world with version control, MRs, pipelines, review process, release, CICD deploy, etc, this is a very slow process with a lot of manual intervention. Some teams just allow the environment to be changed manually through the console and automate scheduled TF plans to detect drift and alert so the values get retroactively updated. This makes me quite nervous.
I dont feel adding ignore_values life-cycle properties everywhere there is a dynamic value as this negates the re-deploy-ability of the service, i.e. on a fresh deployment, the previous value is not considered and just deploys from original defaults.
Ideally, they update the value via some sort of process, potentially a UI or some sort of runbook. This process will update this dynamic value somewhere and then instantly apply it to the environment. This dynamic value is referenced via Terraform also so even if Terraform does run again, it is referencing the same source used during the external process and the plan has no changes.
My thinking is something like version controlled DynamoDB table (Implementing version control using Amazon DynamoDB | AWS Database Blog) with a module to handle the outputs to TF from it and the table management but this has its own issues with making sure that versioning is maintained in other processes, i.e. the runbooks, etc. There is also the lambda route to manage the table contents and retrieval and i have also considered a new provider with variable resources.
Interested in how others have gotten round this.