Celery and Redis with Consul Connect on k8s

Consul
,
1

I’m setting up a Django app with Celery. Redis is the message broker. All three components are running in a separate pod and have consul proxies injected. Celery is unable to connect to the Redis pod:

[2021-08-03 17:28:27,872: ERROR/MainProcess] consumer: Cannot connect to redis://:**@127.0.0.1:6379/1: Error while reading from socket: (104, ‘Connection reset by peer’).
Trying again in 2.00 seconds… (1/100)

Using an unencrypted connection outside of the Consul mesh works as intended.

The Django app can communicate with a separate Postgres and Vault pod over the Consul mesh. This seems to only affect Redis. Could this be a Redis protocol over App Mesh issue?

2

Hi @ahlae,

Can you share the service registration configuration that you’re using for redis? Also, please provide any relevant logs from the Envoy sidecars for the Celery and Redis services which might be helpful in debugging.

Thanks.

3

Hi @blake,

thank you very much for your time! :slight_smile:

Here are the logs:

logs-from-consul-sidecar-in-exp-celery-6ff4c9588-ctkcz.txt (19.7 KB)
logs-from-consul-sidecar-in-exp-redis-master-0.txt (21.5 KB)
logs-from-envoy-sidecar-in-exp-celery-6ff4c9588-ctkcz.txt (75.8 KB)
logs-from-envoy-sidecar-in-exp-redis-master-0.txt (37.9 KB)

Service registration is done via k8s service-sync. Should I extract the service registration configuration from somewhere out of the cluster?

Redis is a plain bitnami/redis chart with a kustomized container name/service account name so that consul can integrate it inside the mesh.

The Celery pod is annotated like this:

‘consul.hashicorp.com/connect-inject’: ‘true’
‘consul.hashicorp.com/connect-service-upstreams’: “exp-redis-master:6379”

Using exp-redis-master:6379 directly in Celery is working.

k8s version is v1.19.6-eks running on EKS. Consul intentions are set to

All Services () Allow All Services ()

Consul image is consul:1.9.4