I am experiencing a weird behaviour with terraform. I have been working on an infra. I have a backend state configured to state my state file in a storage account in azure. Until yesterday everything was fine, this morning when I tried to update my infra, the output from terraform plan was weird as its trying to create all the resources as new, when I checked my local testate…it was empty.
I tried terraform pull and terraform refresh but nothing, still same result. I checked my remote state and I have all the resources still declared.
So I went for plan b, copy and paste my remote state into my local project and run terraform once again, but nothing, seems that terraform is ignoring my terraform state on my local and doesn’t wanna pull the remote one.
Any help please?
Hi,
Weird that the backend got disconnected in your config.
If I understand correctly, the state is empty while it should not be empty and terraform state list returns nothing.
How did you configure the backend in the first place, hard coded in .tf files or with terraform init -backend-config=...?
Depending on how you configured it, you can try to reconfigure like this:
terraform init -reconfigure
This is how I configured my backend
terraform {
backend "azurerm" {
resource_group_name = "<resource-group-name>"
storage_account_name = "<storage-name>"
container_name = "<container-name>"
key = "terraform.tfstate"
}
}
When I run terraform state list the only thing that prints out, are my output everything else is ignored.
Is there any overriding risk if I run terraform init -reconfigure?
The reconfigure command won’t overwrite the remote state, it might clear the local state but that is empty in your case anyways.
But it is a good idea to create backups of your remote state.
You can get the help info like this:
terraform init --help
Usage: terraform init [options] [DIR]
(...)
-reconfigure Reconfigure the backend, ignoring any saved
configuration.
Looking at your backend block, it seems like you already specified everything in that .tf file. The reconfigure command might do nothing then, I’m not entirely sure. If you didn’t hard code for example the container_name and key, it promts you for that when running reconfigure.
I tried to use the reconfigure too but nothing. I even tried to copy and paste my remote state into my local, but terraform just ignores my local tfstate completely.
The local tfstate is ignored because you specified a remote state, that is expected. If you want to use that local state, you need to remove / comment out that backend block.
Any clue what change caused your configuration to stop working? Did you upgrade the terraform version or modified some configuration?
Another thing you could try is making a backup of the working directory (containing your Terraform root module) and remove the .terraform directory and initialize again.
1 Like
The only thing I have done this morning, was to upgrade Lua rock… which I don’t believe has anything to do with terraform.
I tried to remove .terraform and terraform.lock and run init didn’t change anything.
Maybe the connection to Azure fails without reporting an error message.
You could try enabling trace logs (as explained here https://www.terraform.io/docs/internals/debugging.html) and run terraform state list again, then inspect the log file.
Done, the only thing that I can find that is strange is this:
2021/04/06 15:35:48 [TRACE] Meta.Backend: BackendOpts.Config not set, so using settings loaded from backend.tf:2,3-20
2021/04/06 15:35:48 [TRACE] Meta.Backend: using already-initialized, unchanged "azurerm" backend configuration
Meta.Backend: instantiated backend of type *azure.Backend
But the response from my backend is 200 OK
Hm, the response is 200 OK, but it doesn’t return the blob that you expected. Are you sure that the container and blob key are the same?
You said you checked the remote state.
Sorry, I’m out of ideas then.
So I tried to use terraform to create new resources and spin them up. Terraform behaved correctly. Created the resource, pushed the tfstate to my storage account. After terraform completion, I run a terraform plan and it worked, printing that the infrastructure is up to date.
And I am using the same backend configuration (of course changing the name of the blob).
I really have no idea what is happening.
Good to hear that your problem is solved. I can imagine it’s unsatisfying that you still have no clue what was going wrong.
Have a nice day!
Sorry mate is not solved. It works if I create a new project, but still the old one don’t work. But thank you so much for your help and time. I really appreciate it 
Oh, I read it another time, so you essentially created a copy now of the environment, with another key for the remote state, and the state management for that second environment works correctly. But your original environment state is still not working.
Yep, I had to spin up another infra for a project. I created a whole new folder with all the resources and configuration. Runned:
terraform init
terraform plan
terraform apply
And everything when smooth. It did created 7 resources and saved the tfstate to my newly created backend.
But concerned to the other 80 projects I have, nothing…still don’t work
So, the old state files behave differently than new ones, maybe something with metadata? It almost sounds like a change from Azure Blob Storage, but I don’t expect that to be the case.
What happens if you copy the tfstate blob to another blob with a different key and use that in the backend config?
Yes same behaviour. I noticed that even if I copy and paste my remote state into my local, and than run a plan terraform just destroys all the content in my local terraform.tfstate. Is this a normal behaviour?
Thank you so much. This solved my issue!
Side note: while there are some considerations in the case where some sensitive data gets accidentally included in the state file, you may want to consider making sure versioning and / or soft delete are enabled for the bucket you’re using for remote state.
This will give you some additional options in the case where something gets wiped out or corrupted, even if you don’t have a local copy.