According to the documentation, it is mentioned that if the “serial” value in the destination state is higher than the state being pushed, Terraform will prevent the push. A higher serial suggests that data is in the destination state that isn’t accounted for in the local state being pushed.
Could you please clarify the following scenarios:
Scenario 1: What happens if the local state file has a “serial” value of 10, and the remote state file also has a “serial” value of 10?
Scenario 2: What happens if the local state file has a “serial” value of 11, and the remote state file has a “serial” value of 10?
Scenario 3: What happens if the local state file has a “serial” value of 10, and the remote state file has a “serial” value of 11?
Your assistance with this clarification would be greatly appreciated. Thank you!
The general idea here is that Terraform makes two assumptions:
- Each new state snapshot has a higher serial number than any previously seen.
- Once a state snapshot has been published with a particular serial number, no other state snapshot should be published with the same number.
Taking those two principles into account, for your scenarios:
- If local and remote both have serial 10 then the second rule comes into play: Terraform will raise an error unless the local and remote snapshots are equivalent.
- If local has serial 11 and remote has serial 10 then Terraform assumes that local is newer than remote and so will overwrite the remote successfully.
- If local has serial 10 and remote has serial 11 then Terraform will immediately return an error, because the remote snapshot appears to be newer than the local snapshot.
When using any state storage other than Terraform Cloud these checks are all made client-side, and so it’s a best effort consistency check that could potentially be subject to race conditions if someone else tries to push a different snapshot at the same time. If you are making manual changes to state then you should communicate with your coworkers to make sure nobody else creates a new state snapshot – either explicitly with terraform state push or through the normal plan/apply flow – until you are finished your maintenance work.
Terraform Cloud does check the same rules on the server and so there’s some additional protection against pushing snapshots out of order. However, I’d still recommend notifying your colleagues if you are doing manual state maintenance work, because if other operations run concurrently then they might, for example, observe your changes to state before observing corresponding changes to the configuration and therefore end up accidentally planning to undo whatever changes you’ve made in the state.
1 Like
In Scenario 2, after manually adjusting the local serial value of the state file by +1 (11) before pushing it, we observe an overall increase of +2 (12) relative to the original state file. However, when utilizing the -force flag during the push operation, the serial value on the remote state file increments by +1 (11) instead.
Question 1: Why does the serial value increase differently when using push command without -force flag?
Question 2: The documentation advises against using the -force flag. So, how can I ensure that the serial value (11) remains consistent in both the remote and local state files without using the -force flag with the terraform state push command?
An extra increment here suggests that Terraform needed to make some kind of modification to the state snapshot before uploading it, and therefore it incremented the serial again to represent that the new snapshot is different from the one you presented. I don’t know exactly why that would be, but one example would be if the state snapshot you provided was created by an earlier version of Terraform and so its format needed to be upgraded slightly to work with the version you are currently using.
Keeping a local file and a remote file “in sync” is not something this command is designed to support. If you’ve configured a remote backend then Terraform assumes that the remote store is authoritative and that any snapshot you might have on local disk is only a temporary artifact used to perform some kind of manual maintenance.
If you are implementing a nonstandard workflow that involves snapshots on local disk then I think you will have two main options to choose from:
-
Configure Terraform for local state storage so that Terraform will consider your local snapshot as authoritative. Each time you run Terraform, first retrieve the state snapshot from whatever location you’ve saved it. After Terraform completes, save the snapshot wherever the next run will expect to find it.
With this strategy you would essentially be implementing remote state storage yourself outside of Terraform, and so you won’t use Terraform’s remote state management at all
-
Continue using remote state storage, but run terraform state pull after every command that might modify the remote state, to pull the latest snapshot from the remote storage to your local file.
In this case, Terraform manages the remote state storage and you maintain your local snapshot of the state manually as a separate step.
Which of these is most appropriate will depend on what your goal is.
Terraform does not offer any mode where it updates the state both locally and remotely at the same time.
1 Like