Upper limit for consul KV watches

Hi,

Do consul possess an upper limit on number of consul watches?

I am creating 200 watcher and i am getting EOF error repeatedly while firing any consul CLI commands and then disconnection takes place. on consul logs i am seeing “too many files open” error while getting my consul up itself.

for 150 entries i’m good to keep a watch and provide update on an HTTP endpoint.

Setup : 3 node consul agents in server mode were running. configuration file tries to register 200 Key watchers on KVs when one consul agent boots up.

What are you ulimits set to? For system and Consul process?
What does lsof say when you see this?

Thanks mike for response

[rajkumar]# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 31186
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 16384
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 31186
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

for lsof on consul grep gives these results

A. before running an agent
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
0
[rajkumar]# lsof | grep ‘consul’ | wc -l
603

B. with 150 watches these results are:
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
3750
[rajkumar]# lsof | grep ‘consul’ | wc -l
4563

C.
[rajkumar]# lsof | grep ‘consul’ | grep ‘ESTABLISHED’ | wc -l
4040
[rajkumar]# lsof | grep ‘consul’ | wc -l
4881

  1. consul put response when system stops responding
    [rajkumar]# consul kv put new one
    Error! Failed writing data: Put http://127.0.0.1:8500/v1/kv/new: EOF

  2. ‘consul members’
    [rajkumar]# consul members
    Error retrieving members: Get http://127.0.0.1:8500/v1/agent/members?segment=_all: EOF

  3. consul get command error
    [rajkumar]# consul kv get key14 Error querying Consul agent: Get http://127.0.0.1:8500/v1/kv/key14: read tcp 127.0.0.1:55574->127.0.0.1:8500: read: connection reset by peer

  4. consul agent process logs after boot up. I have created 500 watches in config file.

This image shows that i can open 16k files for my consul process as defined by soft limit. then, why keeping more watches is throwing me this issue?

@rajkumarpatel2602 Have you solve this problem, I have exactly met the same one. I have 1000 watch, and there many “EOF” and “connection reset by peer” in log when accessing consul kv. But the ulimit in my container is about 1million. I do not know why it happened.

@jrasell @blake