Consul memory exhausted

Consul
1

Hello,

Vault configured with 2 vault server and 3 consuls. Last week we moved two vault server under a load balancer after this change we have noticed all 3 consul memory (8G per node) exhausted. Also, vault servers are unsealed multiple times. Below are the errors which we noticed during this issue.

    vault[16184]: 2020-07-03T13:04:48.082Z [ERROR] expiration: failed to revoke lease: lease_id=sys/wrapping/wrap/h1bc3e02
 vault[16184]: 2020-07-03T13:04:58.081Z [ERROR] expiration: shutting down, not attempting further revocation of lease:
vault[16184]: 2020-07-03T13:02:55.903Z [INFO]  expiration: revoked lease: lease_id=auth/prod/xxxx/xxx/xxxx/login/************************


vault[16184]: 2020-07-03T13:03:52.012Z [DEBUG] expiration: leases loading: progress=24000
vault[16184]: 2020-07-03T13:04:10.836Z [ERROR] core: failed to create token: error="failed to persist accessor index entry: Unexpected response code: 500 (rpc error making call: rpc error making call: EOF)"

Jul 03 13:04:10  vault[16184]: 2020-07-03T13:04:10.842Z [ERROR] expiration: failed to revoke lease: lease_id=sys/wrapping/wrap/h41 error="failed to read lease entry sys/wrapping/wrap/he1: Unexpected response code: 500"

As a temporary solution, we have doubled the memory size. Does anybody have any solution for the above-mentioned errors?

Thanks

2

Consul maintains all data in memory. How big is your Consul database? You can get an idea by looking at a snapshot.
You can start investigating your data here https://learn.hashicorp.com/vault/monitoring/inspecting-data-consul

Possibly you have something generating a lot of leases with longer TTLs, ie, hundreds of thousands.
Are you seeing anything like this in the log:
[WARN ] expiration: lease count exceeds warning lease threshold

1 Like