I am running HA Proxy in a docker container on Nomad and also a backend service on Nomad. I’m using the consul resolver to allow ha proxy to find the other application. This was working but recently started failing in an odd way: Backend entries show happy for a moment, then report a socket error claiming (entry removed from SRV record)."
From the host on which the ha proxy container runs, dig reliably returns stable IP addresses and ports.
The relevant ha proxy conf is this:
backend dispatcherhttp
balance roundrobin
server-template dispatcherhttp 3 _dispatcherhttp._tcp.service.consul resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check maxconn 20
resolvers consul
nameserver consul 127.0.0.1:8600
accepted_payload_size 8192
hold valid 5s
Has this sort of flakiness been reported before?
If I stop using consul dns and hardcode the IPs things are stable.
Why do you suppose HA Proxy is getting less reliable responses from consul than dig?
Any ideas for what I should try?