We are using a singly-owned item that we store in Consul Key-Value. We want exactly one instance to be able to modify it, but anyone can read it. We accomplished this by using a KV entry with the value containing a JSON blob of some data and the identifier of the instance. Instances gain ownership by doing a consistent read, making sure there’s no owner, and then doing a write using the Check-And-Set value.
The consistent reads has the potential to create a large volume of traffic between individual Consul nodes. One of the considerations was to switch to using Consul Sessions to express ownership of the KV entry (instead of putting it in the JSON payload).
How does the bandwidth compare? Does the consistent read plus write use more bandwidth than attempting to claim a session and writing to the KV store if successful? What about a stale read and a write versus claiming a session?