We are in the process of migrating terraform states to a new backend but have a use case where we need to maintain both the old and the new in order to allow for smaller piece meal migrations instead of having to migrate 170+ states at the same time due to a remote state moving to a new bucket.
Is there a best practice as far as maintaining both states simultaneously as far as allowing changes to propagate to both states in both buckets when a change is made?
Hi @btucker,
Several of Terraform’s expected behaviors rely on the idea that each remote object is tracked by exactly one resource instance in exactly one state, and so Terraform has no built-in mechanisms to intentionally replicate an object into a second state.
It sounds like the crux of your problem is that you are using terraform_remote_state to share data between different configurations and so you’ve run into one of the classic disadvantages of that: it forces using the same artifact for Terraform’s own state tracking as for distributing data to other configurations, and therefore the location of the state for a configuration becomes part of its public interface rather than being an implementation detail.
As a historical sidebar, you can see me learning about that disadvantage and proposing a solution to it at a time when I was using Terraform at another company and not yet working on Terraform at HashiCorp. That discussion is what led to the eventual proposal of generalized data sources and then, some time later after several others encountered similar problems, the recommendation not to use terraform_remote_state.
Of course, it’s not helpful in your situation for me to say "you shouldn’t have used terraform_remote_state" and so although I would recommend planning to transition away from it in the future for this reason, I will focus my comment here on what you might do within your existing design in order to complete your imminent migration.
That terraform_synthetic_state resource I proposed in the PR I linked above was essentially a formalization of how I’d addressed a similar problem myself manually: I wrote a small program that would fetch the Terraform state, extract just the output values portion of it, and construct something that was shaped the same way as a Terraform state but only included the output values. I then published that “state stub” at the old location so that the old referrers could still find it there with terraform_remote_state, but because that “stub” included no resource instances there was no risk of someone trying to use it as a real state for an active configuration and thus violating Terraform’s assumption that a particular object would be managed only in one place.
I don’t have any part of that temporary workaround available to me anymore because it was owned by my old employer and has long ceased to be necessary anyway (we did eventually transition away from terraform_remote_state), but the essence of it was an extra step to be run after terraform apply completes, which uses terraform state pull to fetch the state, some hacky scripting to remove all of the tracked resource instances, and then directly (without using Terraform CLI) write the resulting state stub into the physical location where the other configurations expect to find it.
In our case we were using a mixture of Amazon S3 and Consul to store the states, so that last step of writing the stubs used the Amazon CLI and a small wrapper around the Consul API respectively, because that ended up being simpler than convincing Terraform itself to write state to an arbitrary location. (The terraform_synthetic_state resource would’ve avoided that being a separate step, but that wasn’t accepted for good reasons you can see in the discussion there.)
I’m sorry I don’t have a more polished answer for you here. I hope that the migration you are attempting will move relatively swiftly and so whatever workarounds you build to support the migration process will be short-lived and therefore not create long-term technical debt. I hope this gives you some ideas to start from!
thank you for that incredibly helpful well thought out response with reference material. You are correct in the assumption that it is too late in the process to pivot away at this time. I will certainly take that under advisement and discuss the ramifications of that change with my team for future considerations.
I think the stub state idea accomplishes this goal well and might be a reasonable approach to this scenario.
Thank you for your time and your insight. Both are greatly appreciated.