Raft in Vault uses its own set of TLS certificates, independent of those that the user controls to protect the API port and those used for replication and clustering. These certs get rotated daily, but to ensure that nodes which were down or behind on Raft log replication don’t lose the ability to speak with other nodes, the newly generated daily TLS cert only starts being used once we see that all nodes have received it.
A recent change in this rotation code triggers a panic when the current cert is more than 24h old. This can happen if the cluster as a whole is down for a day or more. It can also happen if a node is unreachable for 24h, or sufficiently backlogged in applying raft logs that it’s more than 24h behind. Finally, all single-node raft clusters older than 48h are also impacted.
Impacted versions: 1.10.1, 1.9.5, 1.8.10 using Integrated Storage. Versions prior to these are unaffected.
New releases addressing this panic are coming soon. Although you should not assume any given Vault version can be downgraded, we’ve examined the changes in these point releases and determined that it should be safe to downgrade to the immediately preceding version - not the last version you were on necessarily, but the last point release preceding these ones - if you can’t wait for the next release.
Thanks for the heads up. We had a migration plan for our non-prod for DR testing and while in DR mode we figured we would switch backends to IS when moving back to the primary site. It shouldn’t bite us but still we’ll hold off on it until the next patch for it.
Hi folks! Thanks for your patience. We’re happy to announce that the fix for this issue has been released! You can find more information about the location of the releases in this GitHub comment on the original issue: