One of our the main motivation for using Nomad is to streamline scheduling and discovery of various monitoring agents, such as Prometheus’ node_exporter for example.
For this, we are using the “system” scheduler which neatly allows to run those agents on every node.
This works as expected, however we still need to instrument the remaining servers (Consul cluster + Nomad servers) with our monitoring agents. The obvious solution seemed to be running Nomad clients alongside Nomad servers however I’m reading in various places this is highly discouraged.
In this use-case, would it be acceptable to run Nomad clients on Nomad server hosts (provided they will be restricted to only run those monitoring agents and nothing else) ?
Another aspect to consider would be your security model. Running an agent in your servers open them to arbitrary execution of jobs (including raw_exec if the plugin is not disabled).
That been said, it is possible and your use case seems to be OK if your monitoring agents are lightweight, but it’s still not something we officially support or recommend.
Preventing other jobs from being scheduled to those clients could also be a bit cumbersome. You could use anti-affinity in your other jobs (negative weights), or maybe set the clients running in the same host as the servers as ineligible for scheduling. Either way, it would require some extra work and coordination to make sure only the system jobs are scheduled in those clients.