I want to understand how Terraform determines that a plan is stale. I am not sure if this is provider dependent, but our backend is in Azure and “stale” appears to be based on the “Last Modified” timestamp of the state blob, even though the content of the state has not changed. I was hoping the plan would only be marked as stale if the content changed. Here is the situation we have:
- We run two workflows simultaneously and both run a plan and apply against the same Terraform configuration, in addition to other configurations that are not shared across workflows.
- When the first workflow is applied, the “shared” configuration has “no changes” and the content of the state file is not changed, but the “Last Modified” timestamp is updated.
- When we run the apply on the second workflow, the “Error: Saved plan is stale.” is thrown, even though the content of the state file has not change.
Is it true that the plan is marked as stale because the plan timestamp is earlier then the “Last Modified” timestamp of the current state file?
Hi @kansberry,
When Terraform creates state snapshots, it includes in them two tracking fields called “lineage” and “serial”. “lineage” is chosen once for the first state snapshot in a series and then preserved for all subsequent snapshots in that series. “serial” is an integer value that increases each time a new state snapshot is created, so any new snapshot should always have a greater serial than its predecessor.
When Terraform saves a plan to disk for applying later, it includes in that plan file the lineage and serial that were associated with the state snapshot that the plan was based on. When you tell Terraform to apply a saved plan, it will retrieve the latest state snapshot and compare its serial and lineage with those recorded in the plan. Both of these tracking fields must be exactly equal between the saved plan and the latest state snapshot at the time you run apply.
“Saved plan is stale” is the error message for when the serial in the plan does not match the serial in the latest state snapshot.
There is another similar error messa ge"Saved plan does not match the given state" which Terraform will return if the lineage does not match between the plan and the latest state snapshot.
Hi @apparentlymart ,
Based on this answer, I ran a plan and apply against a single Terraform configuration. Prior to executing the workflow, I logged into Azure and copied the “serial” and “lineage” values from the blob “state file” associated with this configuration. The plan indicated “no changes” would occur and I then applied this change. In review of the “state file” after the apply, neither the “serial” or “lineage” value changed. Is this what you would expect?
To be clear, when we run our workflows, we do NOT create Azure snapshots of the state file, but I am guessing this was not the “snapshot” you reference in your response. If your response is talking about Azure “snapshots”, since we are not doing this, I am guessing these two fields would not change?
Bottom line, want to understand when these fields would change so that I can better understand when a plan goes “stale”. As of right now, not seeing them change, so plan would never go stale in my scenario, but the “stale” error is being thrown.
If no changes occurred then Terraform has no need to create a new state snapshot, so no new object would be written to your state storage in that case. The object you are looking at is the same object from the previous run.
When I say “state snapshot” I am using Terraform terminology, not Azure terminology. How exactly these concepts will map on to the underlying storage depends on which storage backend you are using, and I’m afraid I’m not personally familiar enough with the Azure backend to know what terms are appropriate for that one.
A “state snapshot” literally means that Terraform Core made an immutable copy of the instantaneous contents of the mutable state structure and sent it to state storage backend. But the state storage backends only actually save the new data if the new content is different than the old content, so a plan that makes no changes cannot create a new state snapshot.