I am running this setup:
- 2 server agents run outside of k8s
- 3 server agents run on k8s nodes + 3 agents running on the same nodes
- 1 non-server agent runs on another k8s node
My understanding is:
- The Helm chart runs both the server StatefulSet and the agent DaemonSet on the same node because of affinity rules limitations.
- The DaemonSet is used so that we can expose the nodePorts using the node IP (and OFC run a single agent, instead of a sidecar container for each service)
If I were able to customize the chart Values (using affinity and node selector rules) so that the DaemonSet wouldn’t run on the nodes running the server StatefulSets, I assume it would still be necessary to expose the non-gossip ports (8500-8502) at host level. Is this correct?
Is there an option to do this easily in the Helm chart or do I need to render the manifests manually and altering them? Could the option be added to the chart values eventually? (Does this makes sense?)