Consul backend tfstate limitations

Hello,

We have this huge project, creating our projects in Openstack and many other things arround. Lately we faced a first issue: the size of our tfstate grew as we add other features until the compressed size was more than the Consul’s default 512Ko limit for the k/v entries.

We raised the limit to 1024, and everything ran fine for a while. Now we face another issue: " Error: Error releasing the state lock Error message: consul lock was lost"

I will probably ask in the Consul section, but I specifically want to know from Terraform developpers/users if raising a new limit is the way to go. Our poor tfstate once extracted is 11Mb large, which is a lot for a json file.

Can we safely try to look for a workaround in Consul’s forum or should we reshape (again) our code to use multiple state, or use another backend ?

Regards
Jerome

When we look at the consul servers logs, we can see messages like this one:

Feb 21 17:19:01 consul-app-dc2-01.node.infra.prx.integ.local consul[538]: 2022-02-21T17:19:01.975+0100 [WARN] agent.server.kvs: Rejecting lock of key due to lock-delay: key=terraform/os-integ-manage-tenants.tfstate/.lock expire_time="2022-02-21 17:19:06.969562798 +0100 CET m=+107966.495768994"

Is the lock expiry duration set by Terraform or is it a Consul setting ?

Thank you