Consul backend tfstate limitations

jbe-dw · February 22, 2022, 1:37pm

Hello,

We have this huge project, creating our projects in Openstack and many other things arround. Lately we faced a first issue: the size of our tfstate grew as we add other features until the compressed size was more than the Consul’s default 512Ko limit for the k/v entries.

We raised the limit to 1024, and everything ran fine for a while. Now we face another issue: " Error: Error releasing the state lock Error message: consul lock was lost"

I will probably ask in the Consul section, but I specifically want to know from Terraform developpers/users if raising a new limit is the way to go. Our poor tfstate once extracted is 11Mb large, which is a lot for a json file.

Can we safely try to look for a workaround in Consul’s forum or should we reshape (again) our code to use multiple state, or use another backend ?

Regards
Jerome

jbe-dw · February 22, 2022, 1:57pm

When we look at the consul servers logs, we can see messages like this one:

Feb 21 17:19:01 consul-app-dc2-01.node.infra.prx.integ.local consul[538]: 2022-02-21T17:19:01.975+0100 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=terraform/os-integ-manage-tenants.tfstate/.lock expire_time="2022-02-21 17:19:06.969562798 +0100 CET m=+107966.495768994"

Is the lock expiry duration set by Terraform or is it a Consul setting ?

Thank you

smutel · March 2, 2022, 2:28pm

github.com/hashicorp/terraform

Error releasing the state lock on consul backend

opened 11:03AM - 02 Mar 22 UTC

closed 01:49PM - 02 Mar 22 UTC

smutel

bug new

### Terraform Version  ``` Terraform v1.1.6 on linux_amd64 ``` ### Terraform Configuration Files  ```terraform terraform { backend "consul" { address = "consul.service.infra.local" scheme = "https" lock = true gzip = true path = "terraform/os-integ-manage-tenants.tfstate" } } ``` ### Debug Output  ### Expected Behavior  No error displayed at the end of terraform plan or terraform apply ### Actual Behavior  ``` No changes. Your infrastructure matches the configuration. Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only plan: terraform apply -refresh-only Releasing state lock. This may take a few moments... ╷ │ Error: Error releasing the state lock │ │ Error message: consul lock was lost │ │ Terraform acquires a lock when accessing your state to prevent others │ running Terraform to potentially modify the state at the same time. An │ error occurred while releasing this lock. This could mean that the lock │ did or did not release properly. If the lock didn't release properly, │ Terraform may not be able to run future commands since it'll appear as if │ the lock is held. │ │ In this scenario, please call the "force-unlock" command to unlock the │ state manually. This is a very dangerous operation since if it is done │ erroneously it could result in two people modifying state at the same time. │ Only call this command if you're certain that the unlock above failed and │ that no one else is holding a lock. ``` ### Steps to Reproduce 1. `terraform init` 2. `terraform plan` ### Additional Context  Here are the consul warning messages at the same time: ``` Mar 2 11:19:40 consul-app-dc2-01.node.infra.local consul[538]: 2022-03-02T11:19:40.488+0100 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=terraform/os-integ-manage-tenants.tfstate/.lock expire_time="2022-03-02 11:19:45.475366862 +0100 CET m=+1382405.001573028" Mar 2 11:20:47 consul-app-dc2-01.node.infra.local consul[538]: 2022-03-02T11:20:47.651+0100 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=terraform/os-integ-manage-tenants.tfstate/.lock expire_time="2022-03-02 11:20:52.629015248 +0100 CET m=+1382472.155221425" Mar 2 11:21:54 consul-app-dc2-01.node.infra.local consul[538]: 2022-03-02T11:21:54.777+0100 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=terraform/os-integ-manage-tenants.tfstate/.lock expire_time="2022-03-02 11:21:59.763651582 +0100 CET m=+1382539.289857795" ``` **Size of the tfstate**: 11M **Delay to do a plan**: ``` real 4m28,765s user 4m13,300s sys 0m22,563s ```