Upper limit for consul KV watches

Consul
1

Hi,

Do consul possess an upper limit on number of consul watches?

I am creating 200 watcher and i am getting EOF error repeatedly while firing any consul CLI commands and then disconnection takes place. on consul logs i am seeing “too many files open” error while getting my consul up itself.

for 150 entries i’m good to keep a watch and provide update on an HTTP endpoint.

Setup : 3 node consul agents in server mode were running. configuration file tries to register 200 Key watchers on KVs when one consul agent boots up.

2

What are you ulimits set to? For system and Consul process?
What does lsof say when you see this?

3

Thanks mike for response

[rajkumar]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31186
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31186
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

for lsof on consul grep gives these results

A. before running an agent
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
0
[rajkumar]# lsof | grep ‘consul’ | wc -l
603

B. with 150 watches these results are:
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
3750
[rajkumar]# lsof | grep ‘consul’ | wc -l
4563

C.
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
4040
[rajkumar]# lsof | grep ‘consul’ | wc -l
4881

  1. consul put response when system stops responding
    [rajkumar]# consul kv put new one
    Error! Failed writing data: Put http://127.0.0.1:8500/v1/kv/new: EOF

  2. ‘consul members’
    [rajkumar]# consul members
    Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: EOF

  3. consul get command error
    [rajkumar]# consul kv get key14 Error querying Consul agent: Get http://127.0.0.1:8500/v1/kv/key14: read tcp 127.0.0.1:55574->127.0.0.1:8500: read: connection reset by peer

  4. consul agent process logs after boot up. I have created 500 watches in config file.

image
image1653×829 657 KB

4

image
image980×555 17.5 KB

This image shows that i can open 16k files for my consul process as defined by soft limit. then, why keeping more watches is throwing me this issue?

5

@rajkumarpatel2602 Have you solve this problem, I have exactly met the same one. I have 1000 watch, and there many “EOF” and “connection reset by peer” in log when accessing consul kv. But the ulimit in my container is about 1million. I do not know why it happened.

@jrasell @blake