I am currently upgrading consul to use the new transparent proxy feature but I am not sure how to properly configure it when using stateful sets.
One stateful set which I can’t get to work at all is redis. The problem is that all communication from clients (other connect services) and redis replicas to the master are all done on the same port (6379). The clients are able to connect to the master via the normal service but the replicas are not able to do it through the headless service. I tried several different configurations via the pod annotations nothing seems to work.
I managed to configure other stateful sets such as kafka since the replicas use a different port then the clients (other consul services) which means I can just exclude the ports used by the replicas by adding
consul.hashicorp.com/transparent-proxy-exclude-outbound-ports which is not ideal either becasue the kafka replicas/broker communication is not encrypted by enovy which means we still need to use the native encryption mechanism provided by kafka.
I was hoping that with the transparent proxy feature consul could also be used to encrypt traffic between replicas of a stateful set but this does not seem to work due to the fact that multple ports are not supported but even if it uses the same port such as with redis it does not seem to work.
Is there a recommendation and/or examples on how to use consul connect with stateful sets. I am really not sure at this point if I am missing something or it just currently due to limitations on how consul connect works.