We have 2 vault servers using Consul KV as backend, they’ve been running for quite a while and we decided to upgrade Consul. After the upgrade, the following occurs; vault will start fine, and unseals, but then shuts itself down with the following being the last error before this happens:
expiration: error restoring leases: error=“failed to read lease entry xxx/issue/client-certificate/I8C5dyfg9QECa0pqP9pAmSQs: Get “http://127.0.0.1:8500/v1/kv/vault/sys/expire/id/xxx/issue/client-certificate/I8C5dyfg9QECa0pqP9pAmSQs”: read tcp 127.0.0.1:52262->127.0.0.1:8500: read: connection reset by peer”
This is then followed by:
Oct 20 12:52:33 vault-001 vault[2832]: 2020-10-20T12:52:33.041Z [ERROR] core: shutting down
Oct 20 12:52:33 vault-001 vault[2832]: 2020-10-20T12:52:33.041Z [DEBUG] core: shutdown called
Oct 20 12:52:33 vault-001 vault[2832]: 2020-10-20T12:52:33.041Z [INFO] core: marked as sealed
Oct 20 12:52:33 vault-001 vault[2832]: 2020-10-20T12:52:33.041Z [DEBUG] core: clearing forwarding clients
Oct 20 12:52:33 vault-001 vault[2832]: 2020-10-20T12:52:33.041Z [DEBUG] core: done clearing forwarding clients
And vault is now sealed. Since our Nomad cluster and much of our other infrastructure depends on this, I’m hoping someone’s got a good answer because we’re about to have the meltdown of all meltdowns happening.