Backend configuration changed (reconfigure and migrate-state don't fix)

Terraform
1

Hi. I’ve followed Jack Tracy’s blog post here and setup an environment that’s been working for weeks but suddenly stopped working with “Backend configuration changed”.

The “Terraform Status Check” works fine with no errors. That runs against a new branch that is not yet merged. The Plan in this pipeline works, so we then proceed to merge into main.

The next pipeline (Plan) is fired CI/CD after the approval to merge. The “terraform init” in that Plan has been failing for the last 3 days & I’m failing to fix it.

I contacted Microsoft Support & after some extensive testing/proving, they are happy that the self hosted Linux agent running the pipelines is running fine.

The error is:
> Error: Backend configuration changed
> A change in the backend configuration has been detected, which may require
> migrating existing state.

The recommended fixes are to run:

  • terraform init -reconfigure and
  • terraform init -migrate-state

I added a task into the Plan pipeline and initially tried reconfigure. That didn’t work, so tried -migrate-state. That hasn’t seemed to work either.

Someone in this Slack forum had the same issue & posted:

> Fixed it by running terraform init -reconfigure and checking the version in the logs and amend as necessary.

Can someone point me in the right direction please to troubleshoot/fix this issue, and explain how to check version (what’s that referring to please) & in what logs.

Looking in the pipeline logs, I see:

2022-05-14T19:43:44.0534820Z 2022-05-14T19:43:44.053Z [TRACE] Preserving existing state lineage “39281718-16af-f366-8d22-b2f300830a28”
2022-05-14T19:43:44.0542246Z 2022-05-14T19:43:44.053Z [TRACE] Meta.Backend: working directory was previously initialized for “azurerm” backend
2022-05-14T19:43:44.0543987Z 2022-05-14T19:43:44.054Z [TRACE] backendConfigNeedsMigration: configuration values have changed, so migration is required
2022-05-14T19:43:44.0545361Z 2022-05-14T19:43:44.054Z [TRACE] Meta.Backend: backend configuration has changed (from type “azurerm” to type “azurerm”)

Many thanks in anticipation.
(NB: My first post here so please be gentle guys!)

2

The blog post you link to mentions the use of the -backend-config flag to terraform init.

If you are supplying some of your backend configuration on the command line, you need to do so every time you run any terraform init command.

The message Backend configuration changed means that you have changed the configuration supplied in your Terraform files backend block or -backend-config CLI options. You should try to understand what change you have made. The record of the existing configuration that Terraform is comparing is found in the .terraform/terraform.tfstate file in your (pipeline’s) working directory.

3

Hi Max,

The backend config hasn’t changed. That was last updated 6 months ago. That’s configured in backend.tf:
image

Putting Terraform Trace on, I get the following:

image
image1181×792 69.2 KB

Which shows the error as:
*** 2022-05-15T19:05:24.670Z [TRACE] Meta.Backend: backend configuration has changed (from type “azurerm” to type “azurerm”)*

Which doesn’t seem to make any sense…?

But I also see: Meta.Backend: merging -backend-config=… CLI overrides into backend configuration

Is that an issue?

BW
Martin

4

As I said:

Since you’ve eliminated the first, it must be the second - apparently you’ve changed the value being passed in the -backend-config CLI option.

It’s a somewhat confusing way of saying the configuration has changed, but the type has not.

5

How do I reset it to say “use the TF state that exists in the .tfstate file”? The contents of the TF state file look fine & unchanged.

BTW: I had changed the terraform.exe version to 1.1.9. I was running 1.1.7.

The pipeline named “Terraform Status Check (Validate & Plan - No Out File)” works fine. That runs with the below init:

terraform init -backend-config=“access_key=$(sttrustukstfazdevops-key2)”

It reads the .tfstate file fine & produces the correct “x to add, 2 to change etc.” So the contents of the state file haven’t changed.

The pipeline named “Terraform Plan” fails. That has the init command:

terraform init -backend-config=“access_key=$(sttrustukstfazdevops-key2)”

i.e. identical., yet fails during “init” with that “backend” error.

Why does the first pipeline not also fail?

Looking at one of the earlier DevOps stages named “Checkout TF-AZDO-Prod@refs/pull/764/merge to s”, I notice that the working pipeline has no errors here.
However this stage in the “Plan” pipeline DOES have numerous errors. e.g.

##[debug]Failed to update oom_score_adj for PID: 24344.
##[debug]System.UnauthorizedAccessException: Access to the path ‘/proc/24344/oom_score_adj’ is denied.

I’m beginning to think therefore that the issue is not with the backend after all, since the first pipeline reads .tfstate exactly as it should.

Thoughts?

6

At this point, I think it must be something wrong with the automation running Terraform - but I know very little about Azure.

Clearly something is different between the pipeline that works and the one that does not. The only things I can think of are:

  • The two pipelines are working with different persistent filesystems - i.e. the contents of the .terraform/ directory are not shared between the pipelines.

  • The command line substitution providing the access key is providing different values.

There is no way to do this, because the .tfstate file when using a remote backend is purely a way for Terraform to alert the user if they have changed things in a way which may require migration to preserve the state.

You can delete the entire .terraform/ directory if you wish, as an alternative way to bypass this. (Though then you remove Terraform’s ability to warn you about accidentally connecting to a different state than you were using before.)

7

This is the one that works:

image
image1737×1199 129 KB

This is the one that fails:

image
image1805×1168 135 KB

So the working directory is the same: terraform (from the “main” Github repo in Azure DevOps).

8

The script that doesn’t work has extra whitespace in it.

9

That was just me testing something.

10

To avoid Key Vault & Secret issues. I “hard coded” the storage account access key into my backend.tf file. After changing the pipeline to just “terraform init”, I then got the below on running the Plan:

image
image1133×489 34.6 KB

I’m running all terraform commands in DevOps pipelines. How can I interactively (i.e. from my laptop) run the -reconfigure or -migrate-state commands? I want to see any output from the commands & respond to any y/n if needed…

11

I opened PowerShell in VScode, navigated to my local clone of the DevOps repo, went into the terraform directory and… ran init fine - no complaints about backend issues. I then successfully ran a Plan too.

That looks then to me as though there is some major issue with the “Terraform Plan” pipeline. I might now contemplate deleting it & recreating it. BTW: I’ve been through it with a toothcomb and it matches.

Thoughts?

12

The thing which confuses me here, is where is the .terraform/ directory being persisted between runs of the pipeline?

If there was no .terraform/ directory, terraform init would not error, as it wouldn’t have any “old” state to compare to and say something has changed.

But, normally, a Git-based pipeline would start with a clean working copy from Git each time…

These two things don’t seem to match up.

13

The state file is remote, I.e. in an Azure storage account.

The “init” successfully connected to that backend state (since I have that backend.tf file).

The “plan” successfully read that state & compared with both Azure itself (to determine changes outside of TF) , & the TF files in the local clone, which is also pushed to the DevOps main repo. Producing exactly my anticipated “7 to add, 12 to change, 2 to destroy”.

14

Hi again. I still have the situation. When I run an “init” in a DevOps pipeline, it always complains of a Backend Configuration Change, so it looks like I do need to fix that.

I’ve tried running -migrate-state from a cmd line within my local clone of the DO repo. The commands work locally & build out the .terraform folder construct & JSON for tfstate too. However that doesn’t fix my DevOps problem. The “init” still fails. So… how do I run the necessary command to fix the issue within DevOps?

Also, I did upgrade the pipelines to run TF 1.1.9 (from 1.1.7). Would that have caused this Backend Configuration Change issue? If so, what should I have done?

Lastly, the .gitignore includes the .terraform folder, so that is not replicated back to the DevOps repo on a push. How does TF know that the backend has changed? Locally within terraform.tfstate I can see lineage/version/serial & hash, which (I believe from research) TF users to determine if there has been a backend change. With no .terraform folder in my DevOps repo, how is TF determining if there has been a change?

Cheers again. Thanks for your help.

15

Somehow, when you run in your pipeline, there is a stale .terraform directory present, it would seem. Add extra debugging to your pipeline to prove this and look at its contents.

16

That’s interesting! I’ve never had a .terraform folder within my DevOps repo. When I set this up I followed Jack Tracey’s blog at the link below. Following that, in my .gitignore is a statement that stops my local repo .terraform folder being copied up to the DO repo.

So first question, should I have a .terraform folder in my DO repo?

I see that if I do a local terraform init, it builds out the .terraform folder structure in my local repo. Should I let that replicate up to the DO repo, by removing */.terraform/ ?

Terraform With Azure DevOps - Jack Tracey - Cloud & Networking

17

No, you’ve got what I was trying to communicate backwards.

I’m saying that somehow - I don’t know how, but I can’t explain the observed behaviour any other way - you do somehow have a pre-existing .terraform directory present in your pipeline when it runs the terraform init command that fails.

(Bear in mind I know nothing about Azure DevOps pipelines so can only extrapolate from general automation concepts.)

1 Like
18

Well, you got me thinking there! I’d had a previous pipeline fail with “Bash exit code 1”. So I wondered if anything got cached on the self hosted agent VM by pipelines. I hadn’t considered this.

I googled it & found that yes caching does occur. It’s also possible to clean out the cache using the “Clean” setting within the pipeline by setting it to “true”. I did that & et-voila - the “terraform init” worked & initialized the backend perfectly fine. I then changed the Clean back to false.

Summary: the issue wasn’t caused by me upgrading to terraform 1.1.9 in the pipeline, nor by the fact that key1 for the storage account had rotated & not updated correctly in the key vault (though I need to sort that too). The cause was exactly as you suspected, a duff .terraform folder must have been left in the pipeline cache.

What an experience/headache! Still… FIXED…! :grinning:

19

The question I now have is, should we have Clean = true" in all Terraform pipelines?..to stop the potential of this ever happening again. And if so, what setting:

  1. Sources
  2. Sources and output directory
  3. Sources directory
  4. All build directories

I think it should true BTW, just not sure which setting.

Many thanks for your help here BTW. Invaluable!

20

These options might make more sense to someone with Azure pipelines experience, but I don’t understand them.

What’s the difference between 1 and 3?

What’s an “output directory” in the context of Terraform? Is the term only there because the options assume a typical software compilation workflow?