Issue connecting to services in remote dc when using federated consul cluster


Setup is as below:

  1. Primary consul cluster on VM ( consul:1.11.1 , envoy:1.18.4 )

  2. Secondary consul cluster on K8S (consul:1.11.1 , envoy:1.18.4 , consul-k8s-control-plane:0.39.0, consul-helm-chart:0.39.0)

  3. Two upstream services deployed on VM

  4. Downstream service deployed on K8S and communicates to the two upstream services using the annotation: “”: “<upstream_service1>:<any_port>:<primary_dc>,<upstream_service2>:<any_port>:<primary_dc>”

The above setup was perfectly running fine for a couple of days… and suddenly stopped working with errors generating in the consul-connect-inject-init container of the downstream service

2022-03-11T21:26:56.563Z [INFO]  Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs.
2022-03-11T21:26:57.566Z [INFO]  Unable to find registered services; retrying
2022-03-11T21:26:58.567Z [INFO]  Unable to find registered services; retrying

Also errors are logged in the consul-connect-injector-webhook-deployment pod as below

2022-03-11T21:22:01.263Z        ERROR   controller.endpoints    Reconciler error        {"reconciler group": "", "reconciler kind": "Endpoints", "name": "<service-name>", "namespace": "<service-namespace>", "error": "1 error occurred:\n\t* upstream \"<upstream_service1>:<any_port>:<primary_dc>\" is invalid: there is no ProxyDefaults config to set mesh gateway mode\n\n"}*Controller).processNextWorkItem

I have ensured while setting up the secondary cluster to deploy the proxy-defaults for it as per this doc and is successfully deployed as well.

Proxy-defaults looks as below:

kind: ProxyDefaults
  name: global
    mode: 'local'

FYI both the upstream services are all healthy in primary consul and running good.

What could be the issue ?

Hi @ashwinkupatkar,

Were you able to resolve this issue? If not, can you share whether the ProxyDefaults entry is showing as having been correctly synced to Consul?

$ kubectl get proxydefaults global


Hi @blake , not sure what caused the issue. After recreating the k8s cluster, the issue went away. We can close this issue.

Thank you.