A reasonable rate limit example on the KV watches


I have experienced a huge consul server performance degradation in a 200-consul-KV-clients cluster. There is a total of 7000+ goroutines in a consul server’s profile. Among them, 1650 goroutines are handling those KV’s block-queries. The CPU of the consul server stayed at its top peak for a very long period.

According to the consul’s online Q/A, I need a rate limit of the KV’s block queries at the client’s side. However, there is no available example of rate-limited methods online. Can you share with me some of your precious advice?