The watches documentations describes how to change a consul agent’s configuration to register a new watch. I assume this requires changing the agent configuration file stored on the agent’s host and then force the agent to reload the configuration.
Is it possible to register a watch through REST APIs? I would like to register a watch remotely, without accessing the machine hosting the Consul agent.
Also, a single agent watching a service’s health can be a single point of failure. If I register the same watch on multiple agents, I assume all of them will execute the associated action (and it will be up to me to synchronize/serialize the executions). Is it correct?
Currently there is no way to configured watches via the API. From a security perspective it would be similar to allowing script checks to be registered via the API which in most cases you wouldn’t want to allow.
You are right that watches executed by an agent would be a single point of failure and that if you add the watch to multiple agents they will all execute it.
There are a number of tools that can watch for changes and do things envconsul, consul-template and the builtin consul watch command. However none of these currently solve the problem of being able to coordinate with other instances and only have one execute the watch at any given time.
The Application Leader Election with Sessions Guide goes over how the API can be used for leader election to ensure only the leader performs the work while any other instances are around in case the leader fails.
If you have any Go development experience the Consul API client package provides facilities for doing the leader election as well as facilities to execute watches.
Also feel free to open a feature request with the details of your use case at github.com/hashicorp/consul.