Consul-Connect fails with StatefulSets

Consul
,
1

I just got started today.

It seems Consul is FAILING due to StatefulSet and headless service. The application I am using has 2 StatefulSets (which has 2 headless services) and 2 regular services (ClusterIP), but this causes issues, and I cannot find docs on this, output form consul-connect-inject-init container:

2022-09-22T05:49:16.015Z [ERROR] There are multiple Consul services registered for this pod when there must only be one. Check if there are multiple Kubernetes services selecting this pod and add the label `consul.hashicorp.com/service-ignore: "true"` to all services except the one used by Consul for handling requests.

The envoy-sidecar then says:

Failed to load logs: container "envoy-sidecar" in pod "dgraph-dgraph-alpha-0" is waiting to start: PodInitializing
Reason: BadRequest (400)

STEPS TO REPRODUCE 

# install consul
# docs https://www.consul.io/docs/k8s/installation/install
helm install consul hashicorp/consul \
  --create-namespace \
  --namespace consul \
  --set global.name=consul \
  --set connectInject.enabled=consul \
  --set controller.enabled=consul

cat << EOF > dgraph_config.yaml
image:
  tag: v21.03.2
zero:
  extraAnnotations:
    consul.hashicorp.com/connect-inject: 'true'
    consul.hashicorp.com/transparent-proxy: 'true'
alpha:
  extraAnnotations:
    consul.hashicorp.com/connect-inject: 'true'
    consul.hashicorp.com/transparent-proxy: 'true'
  configFile:
    config.yaml: |
      security:
        whitelist: {{ env "DG_ACCEPT_LIST" | default "0.0.0.0/0" | quote }}
EOF 
helm repo add dgraph https://charts.dgraph.io
helm install dgraph dgraph/dgraph --values dgraph.yaml

What do I need to do? I cannot find docs on StatefulSet vs. Consul-Connect.

2

This is definitely a bug, as StatefulSet + headless service is a standard feature of Kubernetes and thus not an uncommon use case.

In the interim, I found a workaround to get at least partial functionality with CSM (Consul Connect) is to use service labels on the headless service. I added the label consul.hashicorp.com/service-ignore: 'true' to the headless services. Now I get some functionality.

The internal communication between cluster nodes are not encrypted, so for this solution, one has to either use another service mesh solution, or manually encrypt the internal traffic.

3

Anyone interested in this?

I managed to get injection to work, but then connectivity does not work. Now sure what to do to debug further.