Consul snapshot size increased in 1.10

Consul
1

Hello, recently we upgraded to Consul 1.10.3 from 1.9.4 and noticed that snapshots have grown anywhere from 50-100% in size on each instance. As far as I know the data being entered into Consul hasn’t grown to account for this. One of the servers in particular has no activity, but the snapshots have grown. We’re using the official Helm chart v0.34.1.

Does anyone know if additional data is now backed up by snapshots, or some other reason they may have grown like this? For example, on 10/5 our snapshots were consuming 65mb each. Today those are 125 mb.

Thanks,
Weldon

2

Are you snapshotting each of your consul nodes?? You shouldn’t. They’re all the same data at different stages. You should only backup the active node.

As far as the increase, you can use:

consul snapshot inspect <file>

To check what is the cause of the increase, it shows you details on what is using up storage in good detail.

3

Thank you very much @aram , that command helped a lot - with the -kvdetails option. I was able to determine it was actually the KVS. I hadn’t taken into account that we’re using Vault and the vault/sys key had grown in size.

Thanks again,
Weldon