Transparent proxy with stateful sets

Consul
,
1

Hey guys,

I am currently upgrading consul to use the new transparent proxy feature but I am not sure how to properly configure it when using stateful sets.

One stateful set which I can’t get to work at all is redis. The problem is that all communication from clients (other connect services) and redis replicas to the master are all done on the same port (6379). The clients are able to connect to the master via the normal service but the replicas are not able to do it through the headless service. I tried several different configurations via the pod annotations nothing seems to work.

I managed to configure other stateful sets such as kafka since the replicas use a different port then the clients (other consul services) which means I can just exclude the ports used by the replicas by adding consul.hashicorp.com/transparent-proxy-exclude-inbound-ports and consul.hashicorp.com/transparent-proxy-exclude-outbound-ports which is not ideal either becasue the kafka replicas/broker communication is not encrypted by enovy which means we still need to use the native encryption mechanism provided by kafka.

I was hoping that with the transparent proxy feature consul could also be used to encrypt traffic between replicas of a stateful set but this does not seem to work due to the fact that multple ports are not supported but even if it uses the same port such as with redis it does not seem to work.

Is there a recommendation and/or examples on how to use consul connect with stateful sets. I am really not sure at this point if I am missing something or it just currently due to limitations on how consul connect works.

Bests regards,
Nico

2

Any consul expert (e.g. @kschoche, @lkysow) has an idea about this topic and can provide some insights? Some feedback would be much appreciated :slight_smile:

3

Hi @nflaig - You mentioned that the redis service has 2 services and both are pointing to the same port, do they also select the same pods as well?

If so, this is also an issue within the current architecture as we dont have a way to support having multiple k8s services registered with consul on the same pod.

Can you confirm if that is the configuration that you’re describing?

4

Hi @kschoche, yes both the normal and headless service use the same port and select the same pods. I linked the two services above, just in case if you want to take a look yourself.