Can I avoid server and agent to run on the same k8s node?

Consul
,
1

I am running this setup:

  • 2 server agents run outside of k8s
  • 3 server agents run on k8s nodes + 3 agents running on the same nodes
  • 1 non-server agent runs on another k8s node

My understanding is:

  • The Helm chart runs both the server StatefulSet and the agent DaemonSet on the same node because of affinity rules limitations.
  • The DaemonSet is used so that we can expose the nodePorts using the node IP (and OFC run a single agent, instead of a sidecar container for each service)

If I were able to customize the chart Values (using affinity and node selector rules) so that the DaemonSet wouldn’t run on the nodes running the server StatefulSets, I assume it would still be necessary to expose the non-gossip ports (8500-8502) at host level. Is this correct?

Is there an option to do this easily in the Helm chart or do I need to render the manifests manually and altering them? Could the option be added to the chart values eventually? (Does this makes sense?)

2

Hi! What features of the chart are you using? e.g. service mesh or catalog sync?

Some of the components assume that there are consul clients on every node so you’d also want to set their affinities so those components don’t land on nodes without Consul clients.

If there’s no components running that are trying to talk to consul clients on the nodes where the servers are then you wouldn’t need hostPorts for the servers on ports 8500-8502 because nothing would be talking to them.

3

Thanks for the quick reply!

This probably answers my question.

Currently I am using only catalog sync, but I will probably use the service mesh functionality in the future, also on the server nodes.

Having the need of running the agent twice is really annoying :sweat_smile:

It would be nice (and probably not a huge effort) having an option like exposeAllPorts on the server values.