Some information of our cluster first (in case this can help):
- Self hosted
- Currently have 16 nodes (stills growing)
- Created using rancher, and we have multiple project in it, with a few namespace per projet. Each project uses a separate consul instance in order to fully isolate the workload.
- We are in a micro service workload
Currently we are using the Consul agent per pod architectures as this was the easiest for us. However as every project have between 10-20 services, with at least two replica. This generate between 20 and 40 consul agents nodes. We see that the documentation here (https://www.consul.io/docs/k8s/installation/overview) mention that we are not in the preferred type of configuration and that it is recommended to use DaemonSet instead.
However this raises a few questions for us:
1 - As we are in a multi-tenant and we want to keep the isolation between all the project workload, if we switch to DaemonSet, we will have a clash of port as each consul cluster will try to bind port 8500 for their agent. I guess we could overwrite the port per project, but is it the advised solution?
2 - We fear that we will end up in the same type of issue later when the cluster is bigger. Currently we would go from 20-40 agent to a static value of 16 (as we got 16 k8s nodes at the moment). But this advantage on resources will quickly go down the more K8S nodes we got.
Do you have any advise on what would be the best practice for our type of installation/workload?