Up the txn_max_req_len from 512KB

Consul
1

I have a need to increase the default limit of 512KB as the zipped terraform state fie is about 950KB. What should txn_max_req_len be set to? Is the vault in Bytes or KB?

2

Hi @jayshah11,

Welcome to the forums, and thank you for posting!

Can you provide more information on how you are using Consul? I don’t want to make assumptions, but is this specifically for using Consul as a Terraform back end?

Looking forward to your reply!
Jono

3

Hello @jsosulska

Yes, I am using consul’s KV engine to store Terraform states. We have been storing zipped files but there is a need to increase the kv size from the default 512KB. My question was if the vault should be in bytes of KB in the consul configuration.

4

Hi @jayshah11,

Since Consul stores terraform in the KV store, have you tried tuning kv_max_value_size in your configs? This will allow you to adjust the KV values, rather than txn_max_req_len which focuses on transactions. The size is in KB, not bytes.

Give this a shot, and let me know how that works for you :slight_smile:

Best, Jono

5

Thank you @jsosulska

It looks like kv_max_value_size expects a value of bytes. The default being 524288 bytes. I bumped it up to 1024000 bytes.

I see that the documentation mentions the change may impact leadership stability. Do we have a solid confirmation that the change impacts any Consul functionality?

6

Hi @jayshah11,

Thanks for the feedback, I did some digging, and here’s what I found:

  • These sizes change how Consul can handle Raft connections, which is the core consensus protocol for Consul. 512kb is the size of the messages used in raft, and as such, on very busy clusters, it may take longer to complete a transaction.
  • In a busy cluster with many clients and servers this leads to raft transaction apply delays, and thus could eventually cause the leader to fail under the load.

These values were implemented as solutions for specific usecases, like using Consul as a backend for Vault. Generally, adjusting these values falls outside of most common deployments.

I would like to hear more about your usecase for using Consul as a storage backend. As your infrastructure scales up, your needs may end up growing past Consul (or may be there already).

I hope this helps!

7

Hi jsosulka,

I am using consul as a terraform backend as recommended in the Hashicorp documentation. We run terraform to build our cloud infrastructure. The size of the terraform state files are growing due to increasing number of hosts and resources. And this is the case even after we enable gzip.

I am happy to discuss it over phone and am open to suggestions. But at this point, it looks like we have to look for other options besides Consul.

I have another issue with using Consul as a backend to Vault and I will post that in the chat and take everyone’s feedback. I understand that Vault does not need Consul anymore with the 1.4 release.

Thanks,
Jay

8

Hi @jayshah11 ,

Thanks for your response, and apologies for the delay in mine! Please see responses inline:

I am using consul as a terraform backend as recommended in the Hashicorp documentation.

Can you provide a link to what specific documentation you’re using? Most of the documentation on backends I’ve seen says local or remote. If there is documentation out there around Consul as a backend, we would want to update it to take into account constraints.

I am happy to discuss it over phone and am open to suggestions. But at this point, it looks like we have to look for other options besides Consul.

Thanks for offering! We’re always looking for ways to improve our products. :slight_smile:

I understand that Vault does not need Consul anymore with the 1.4 release.

This is correct. Vault supports a number of storage backends, most recently integrated storage. This means any storage backend is now optional.

I hope this helps, and thank you again for your patience in my reply.