Backend configuration changed (reconfigure and migrate-state don't fix)

Hi. I’ve followed Jack Tracy’s blog post here and setup an environment that’s been working for weeks but suddenly stopped working with “Backend configuration changed”.

The “Terraform Status Check” works fine with no errors. That runs against a new branch that is not yet merged. The Plan in this pipeline works, so we then proceed to merge into main.

The next pipeline (Plan) is fired CI/CD after the approval to merge. The “terraform init” in that Plan has been failing for the last 3 days & I’m failing to fix it.

I contacted Microsoft Support & after some extensive testing/proving, they are happy that the self hosted Linux agent running the pipelines is running fine.

The error is:
> Error: Backend configuration changed
> A change in the backend configuration has been detected, which may require
> migrating existing state.

The recommended fixes are to run:

  • terraform init -reconfigure and
  • terraform init -migrate-state

I added a task into the Plan pipeline and initially tried reconfigure. That didn’t work, so tried -migrate-state. That hasn’t seemed to work either.

Someone in this Slack forum had the same issue & posted:

> Fixed it by running terraform init -reconfigure and checking the version in the logs and amend as necessary.

Can someone point me in the right direction please to troubleshoot/fix this issue, and explain how to check version (what’s that referring to please) & in what logs.

Looking in the pipeline logs, I see:

2022-05-14T19:43:44.0534820Z 2022-05-14T19:43:44.053Z [TRACE] Preserving existing state lineage “39281718-16af-f366-8d22-b2f300830a28”
2022-05-14T19:43:44.0542246Z 2022-05-14T19:43:44.053Z [TRACE] Meta.Backend: working directory was previously initialized for “azurerm” backend
2022-05-14T19:43:44.0543987Z 2022-05-14T19:43:44.054Z [TRACE] backendConfigNeedsMigration: configuration values have changed, so migration is required
2022-05-14T19:43:44.0545361Z 2022-05-14T19:43:44.054Z [TRACE] Meta.Backend: backend configuration has changed (from type “azurerm” to type “azurerm”)

Many thanks in anticipation.
(NB: My first post here so please be gentle guys!)

The blog post you link to mentions the use of the -backend-config flag to terraform init.

If you are supplying some of your backend configuration on the command line, you need to do so every time you run any terraform init command.

The message Backend configuration changed means that you have changed the configuration supplied in your Terraform files backend block or -backend-config CLI options. You should try to understand what change you have made. The record of the existing configuration that Terraform is comparing is found in the .terraform/terraform.tfstate file in your (pipeline’s) working directory.

Hi Max,

The backend config hasn’t changed. That was last updated 6 months ago. That’s configured in backend.tf:
image

Putting Terraform Trace on, I get the following:

Which shows the error as:
*** 2022-05-15T19:05:24.670Z [TRACE] Meta.Backend: backend configuration has changed (from type “azurerm” to type “azurerm”)*

Which doesn’t seem to make any sense…?

But I also see: Meta.Backend: merging -backend-config=… CLI overrides into backend configuration

Is that an issue?

BW
Martin

As I said:

Since you’ve eliminated the first, it must be the second - apparently you’ve changed the value being passed in the -backend-config CLI option.

It’s a somewhat confusing way of saying the configuration has changed, but the type has not.

How do I reset it to say “use the TF state that exists in the .tfstate file”? The contents of the TF state file look fine & unchanged.

BTW: I had changed the terraform.exe version to 1.1.9. I was running 1.1.7.

The pipeline named “Terraform Status Check (Validate & Plan - No Out File)” works fine. That runs with the below init:

terraform init -backend-config=“access_key=$(sttrustukstfazdevops-key2)”

It reads the .tfstate file fine & produces the correct “x to add, 2 to change etc.” So the contents of the state file haven’t changed.

The pipeline named “Terraform Plan” fails. That has the init command:

terraform init -backend-config=“access_key=$(sttrustukstfazdevops-key2)”

i.e. identical., yet fails during “init” with that “backend” error.

Why does the first pipeline not also fail?

Looking at one of the earlier DevOps stages named “Checkout TF-AZDO-Prod@refs/pull/764/merge to s”, I notice that the working pipeline has no errors here.
However this stage in the “Plan” pipeline DOES have numerous errors. e.g.

##[debug]Failed to update oom_score_adj for PID: 24344.
##[debug]System.UnauthorizedAccessException: Access to the path ‘/proc/24344/oom_score_adj’ is denied.

I’m beginning to think therefore that the issue is not with the backend after all, since the first pipeline reads .tfstate exactly as it should.

Thoughts?

At this point, I think it must be something wrong with the automation running Terraform - but I know very little about Azure.

Clearly something is different between the pipeline that works and the one that does not. The only things I can think of are:

  • The two pipelines are working with different persistent filesystems - i.e. the contents of the .terraform/ directory are not shared between the pipelines.

  • The command line substitution providing the access key is providing different values.

There is no way to do this, because the .tfstate file when using a remote backend is purely a way for Terraform to alert the user if they have changed things in a way which may require migration to preserve the state.

You can delete the entire .terraform/ directory if you wish, as an alternative way to bypass this. (Though then you remove Terraform’s ability to warn you about accidentally connecting to a different state than you were using before.)

This is the one that works:

This is the one that fails:

So the working directory is the same: terraform (from the “main” Github repo in Azure DevOps).

The script that doesn’t work has extra whitespace in it.

That was just me testing something.

To avoid Key Vault & Secret issues. I “hard coded” the storage account access key into my backend.tf file. After changing the pipeline to just “terraform init”, I then got the below on running the Plan:

I’m running all terraform commands in DevOps pipelines. How can I interactively (i.e. from my laptop) run the -reconfigure or -migrate-state commands? I want to see any output from the commands & respond to any y/n if needed…

I opened PowerShell in VScode, navigated to my local clone of the DevOps repo, went into the terraform directory and… ran init fine - no complaints about backend issues. I then successfully ran a Plan too.

That looks then to me as though there is some major issue with the “Terraform Plan” pipeline. I might now contemplate deleting it & recreating it. BTW: I’ve been through it with a toothcomb and it matches.

Thoughts?

The thing which confuses me here, is where is the .terraform/ directory being persisted between runs of the pipeline?

If there was no .terraform/ directory, terraform init would not error, as it wouldn’t have any “old” state to compare to and say something has changed.

But, normally, a Git-based pipeline would start with a clean working copy from Git each time…

These two things don’t seem to match up.

The state file is remote, I.e. in an Azure storage account.

The “init” successfully connected to that backend state (since I have that backend.tf file).

The “plan” successfully read that state & compared with both Azure itself (to determine changes outside of TF) , & the TF files in the local clone, which is also pushed to the DevOps main repo. Producing exactly my anticipated “7 to add, 12 to change, 2 to destroy”.