Consul Connect Root CA rotation process

Hi, we’ve been using Consul Connect with Vault as an external CA, and have provided Consul servers with a periodic token to manage the PKI endpoints completely with the permissions specified in the docs.

However on Saturday, the root CA in Vault seems to have expired without Consul replacing or renewing it properly.

My question is whether Consul is supposed to handle this process or if we’re expected to provide a new PKI endpoint before the root expires?

Hey @sgtang

We support gradually rotating root CA with Vault. These docs describe the process.

Hi, thank you for the docs. We’ve been trying to use the rotation process via consul connect ca set-config command, just changing to a new Vault root endpoint with a different name.

We see the CA roots update in Consul and existing proxies work fine, however any new proxies we start will error out with "Failed to load trusted CA certificates from " until we specifically restart the Consul leader and an election is forced.

I haven’t found any docs/issues regarding this behavior. It seems like a bug but we’re not entirely sure. Are we implementing this incorrectly?

Hmm it could be. Would you mind creating an issue in hashicorp/consul-helm with some reproduction steps and we can confirm there?

We actually resolved this issue by upgrading from 1.9.6 to 1.9.8, this thread here covers the details: Secondary Datacenter can't load trusted CA certs after Rotation (Vault CA) · Issue #10682 · hashicorp/consul · GitHub

Thanks for your help