Is there a strategy for dealing with state migrations in an environment of multiple developers and version control branches?

Terraform
1

So. The scenario is that we want to change the names of the resources in Terraform. Nothing major, just some clean up/consistency issues.

Using terraform state mv we can rename the resources, run the plan and see no changes. We are happy.

This scenario works perfectly if you are the only developer making changes or every change is direct to the main branch in version control.

What is a more realistic scenario is that one of the developers has been tasked with making the changes, so they have a feature branch in the version control. They cannot (I hope obviously) make changes to the tfstate as that would be inconsistent with the main branches code which may be needed for a release at any time.

In other projects, the mechanism to handle this scenario is called ‘migrations’. This is where the developer can describe what they want to change (and how to reverse it) as part of their branch. The migration system looks after making sure that all changes are executed in a consistent order, etc. (traditionally used for amending a database, but, in theory, can be used for anything).

Terraform migrations would be a really nice feature to help migrate the state for a branch, so prior to executing the plan, all outstanding migrations are run. In the event of a failure, the migrations are “rolled back” (hence the need to either have a mechanism of describing how to reverse a migration or a structure for reversible migrations).

In the projects where we’ve had migrations, a branch to be merged may only contain migrations that need to be run AFTER a deployment. We found that simple blocking tasks in project management dealt with this far easier that attempting to describe ‘before’ and ‘after’ migrations (though, of course, if there is an easy pattern for this, then fine).

Another possibility for this COULD be to have a non applying change to the state purely for planning.

So, the goal is, in a branch, a develop can rename the resources, run the plan and see the plan result to no changes (so indicating the rename is correctly done and resources are not being deleted/created because of a new name), but the rename hasn’t been applied to the live state (so the main branch will still see a fully valid state as it applies to the current resources and the .tf code in the main branch).

Think of it as a try-before-you-buy thing. Modify state, plan, see the results, throw it all away if not happy.

Basically, any ideas would be really appreciated.

2

Hi @rquadling,

I’m afraid this is a bit of a “good news, bad news” situation…

The bad news is that Terraform currently has no mechanism for this other than terraform state mv, which as you noted is problematic when coordinating changes for shared modules that are used by various callers that might all upgrade at different times.

The good news is that the Terraform team is currently working on something very much like what you described here, and we’re hoping to complete it for the next minor release. You can read more about this in the first draft of the Refactoring docs; the details may still shift before final release but the overall principles of it are settled now and implementation is already underway.

We also made analogy to database migrations while initially thinking about this, although ultimately the final design ended up somewhat simpler than database migrations because Terraform has the advantage that it can handle the refactoring statements at the same time as it’s planning other changes, and so there isn’t the typical problem with database migrations that there’s a separation between deploying the stateless compute portion of a new release and deploying the stateful data store portion.

As I think you were imagining, the idea would be that you’d just run terraform apply or terraform plan as normal and Terraform would in effect behave as if the objects had originally been named under their new names, even though the previous run state used the old names. Terraform will note the address changes in the plan UI for your reference but won’t require any additional actions on your part once you’ve added the relevant statements to the configuration.

3

Hi @apparentlymart

From what I’ve read on the refactoring docs draft, I’m impressed. The approach described certainly feels VERY comfortable to use.

Keeping the history of moved resources is critical. In my database analogy, the schema changes can mount up quite a lot and so we occasionally “reset” the base schema (essentially roll up all the migrations into a new schema and archive the migrations).

This works well for us as it is our application and database, but I can’t see this being a particularly useful strategy for Terraform module writers as they are not responsible for the current state. For purely local users, I suspect once the move has taken place, then the moved blocks could be seen as redundant. Certainly keeping them in the version control repository should be the norm.

It does look like the approach described is a forward only approach, so going back to the previous commit in a version control repository that holds the terraform structures, the undoing of the move wouldn’t be possible.

We have a “rollback” strategy within our database migrations. Some argument exists on if we should ever “rollback” but rather “fix forward” instead.

I suppose with suitable planning and testing, “rollback” should only ever be a last resort. In this scenario with Terraform, the from and to values in the moved block would need to be swapped.

I can’t think of any mechanism that this would be able to operate automatically as Terraform itself has no concept of “reversing” a plan to a previous state (though you can do so if you have a versioning backend I suppose but that is starting to feel VERY hacky and just increases the amount of trouble you could be in if things go wrong with that). But even that ends up with problems as the previous state would still need the reversed moved block from and to to make sure resources were renamed from their new place to their old place that the previous state file would know about. All starts getting VERY messy.

But, certainly looking forward to this new feature. In the meantime, I’m carefully documenting the terraform state mv when this is a requirement.