We have Consul Primary DC and Consul Secondary DCs connected together and running. Each of the Consul DCs runs on a separate GKE cluster, and they are connected using Consul Mesh Gateways. TLS is enabled to secure communication between Consul Servers and Clients. For Consul Clients we use the operator method to deliver certificates to them (this is done automatically by the Consul Helm Chart). For now we are using Consul build-in CA which was initially generated in the Consul Primary DC upon its bootstrap. Then for each Consul Secondary DC the CA certificate and the corresponding private key were copied using the Consul federation secret (according to Consul documentation). We are not using Consul integration with Vault for now.
The question is:
How to rollout the CA root certificate used to protect the TLS communication between Consul Agents? (I am not asking about rolling certificates for Consul Servers or Consul Clients).
The next question is: How to perform the rollout of the CA root certificate across Consul Primary DC and Consul Secondary DCs?
For Consul Primary DC:
I know that on Kubernetes the CA certificate and its corresponding private key are stored as Kubernetes secrets (these are used to generate certs for all Consul Agents).
For Consul Secondary DC:
I know that on Kubernetes the CA certificate and its corresponding private key are stored as the Consul Federation secret exported from Consul Primary DC (these are used to generate certs for all Consul Agents).
The procedure should guarantee zero-downtime for Consul Clients/Servers, Consul k8s components and services running in the Consul service mesh.
Thanks for any help in advance.
Dominik