K8s - consul-k8s-control-plane connect-init fails on ephemerals pods since 1.10 upgrade

Consul
,
1

Hello,

We upgraded consul from 1.9 to 1.10 on federated datacenters and we now get some issues in consul-connect-inject-init init container on pods which does not have service defined.
When it tries the command :
consul-k8s-control-plane connect-init -pod-name=${POD_NAME} -pod-namespace=${POD_NAMESPACE}
we get this errors :

2022-02-08T13:08:42.831Z [INFO]  Unable to find registered services; retrying
2022-02-08T13:08:43.833Z [INFO]  Unable to find registered services; retrying
2022-02-08T13:08:44.835Z [INFO]  Unable to find registered services; retrying
2022-02-08T13:08:45.836Z [INFO]  Unable to find registered services; retrying
2022-02-08T13:08:45.836Z [INFO]  Check to ensure a Kubernetes service has been created for this application. If your pod is not starting also check the connect-inject deployment logs.
2022-02-08T13:08:46.856Z [INFO]  Unable to find registered services; retrying
2022-02-08T13:08:46.856Z [ERROR] Timed out waiting for service registration: error="did not find correct number of services: 0"

If I create a service for this pod (which don’t need one), it works well.

Pod is using theses annotations :

consul.hashicorp.com/connect-inject: true
consul.hashicorp.com/connect-service-upstreams: haproxy-webhook:11111:azr-pilot-francecentral

We did not have this behavior with 1.9 version, and a consul service was auto created, we did not use consul-k8s-control-plane but consul-k8s.

Current versions are :

  • hashicorp/consul:1.10.4
  • hashicorp/consul-k8s-control-plane:0.33.0
  • envoyproxy/envoy-alpine:v1.17.4

Maybe we missed a configuration in chart for that ?

Thanks

2

Hi @arkpoah! Thanks for filing this issue.
What you are experiencing is a breaking change that occurred when we released Consul 1.10 which requires a Kubernetes Service for every Consul service. There is a blurb about this in the Changelog as well as the documentation which I attached to this post.

If you do not need to expose your Consul service through any Kubernetes service ports then it should be sufficient to deploy a headless k8s service.

3

Yes thanks, it seems good with a headless k8s service.
We did not see this breaking change because we looked at consul releases page instead of consul-helm.