Consul mesh federation and kubernetes API

Hi, We are working on consul meshes on kubernetes clusters across two different data centers which need to be federated. While going through the doc, we noticed that, the primary DC consul servers should be able to talk to the secondary DC kubernetes API for token verification.

it would be really helpful, if someone can help me to understand why this is needed? All the API requests will be handled by primary DC consul servers only? And what happens if primary DC is down? Services in secondary consul mesh will continue to operate ? Please advice

Hi @hansemmanuel !
The primary dc needs to reach the secondary dc’s Kubernetes API so that it can verify the identify of the secondary’s components when issuing global scoped ACL tokens. Once those tokens are provisioned during the startup of the component pod there shouldnt be any additional calls to the secondary kube api, until you restart those pods.

1 Like

@kschoche Thanks for the clarification. So it means that, in federated mesh, primary consul will be always issuing ACL tokens? and with this approach, if primary DC is down, then if some of the services in mesh is restarted, it wont be able to join back to the mesh until the primary is up and running again?

Hi @hansemmanuel this is only the case for global scoped ACL tokens as those are always issued from the primary datacenter. Local scoped ACL tokens (like mesh services) in a secondary should not be affected by the primary being down.

Thanks for the clarification.

It would be great to include a diagram describing how the k8sauth method works in federation mode in the docs.
After a week of trouble in upgrade from 0.40.0 to 0.43.0, reading the comments in helm values.yaml. I first learned that primary have to connect to secondary API server.

Hi @manobi! Thanks for bringing that up, adding a diagram to the federation docs is a great idea and it is on our list of todos. Stay tuned!