Hello,
Setup is as below:
-
Primary consul cluster on VM ( consul:1.11.1 , envoy:1.18.4 )
-
Secondary consul cluster on K8S (consul:1.11.1 , envoy:1.18.4 , consul-k8s-control-plane:0.39.0, consul-helm-chart:0.39.0)
-
Two upstream services deployed on VM
-
Downstream service deployed on K8S and communicates to the two upstream services using the annotation: “consul.hashicorp.com/connect-service-upstreams”: “<upstream_service1>:<any_port>:<primary_dc>,<upstream_service2>:<any_port>:<primary_dc>”
The above setup was perfectly running fine for a couple of days… and suddenly stopped working with errors generating in the consul-connect-inject-init container of the downstream service
2022-03-11T21:26:56.563Z [INFO] Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs.
2022-03-11T21:26:57.566Z [INFO] Unable to find registered services; retrying
2022-03-11T21:26:58.567Z [INFO] Unable to find registered services; retrying
Also errors are logged in the consul-connect-injector-webhook-deployment pod as below
/home/circleci/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227
2022-03-11T21:22:01.263Z ERROR controller.endpoints Reconciler error {"reconciler group": "", "reconciler kind": "Endpoints", "name": "<service-name>", "namespace": "<service-namespace>", "error": "1 error occurred:\n\t* upstream \"<upstream_service1>:<any_port>:<primary_dc>\" is invalid: there is no ProxyDefaults config to set mesh gateway mode\n\n"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/circleci/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/home/circleci/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.2/pkg/internal/controller/controller.go:227
I have ensured while setting up the secondary cluster to deploy the proxy-defaults for it as per this doc and is successfully deployed as well.
Proxy-defaults looks as below:
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
name: global
spec:
meshGateway:
mode: 'local'
FYI both the upstream services are all healthy in primary consul and running good.
What could be the issue ?