Consul-template - rpc error: code = InvalidArgument desc = Key is required

Consul
1

Hello everyone!

I need some insight related to a consul-template issue. I have a loadbalancer (HAproxy) with a configuration rendered via consul-template. Everything works fine but there some issue visible in the logs I can’t wrap my head around.

Here is my Consul-template HAproxy template:


global
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

defaults
    log global
    mode    http
    balance source

    option httplog
    option dontlognull
    option log-separate-errors
    option httpclose
    option abortonclose
    option http-server-close
    option redispatch

    timeout connect 5000
    timeout client  50000
    timeout server  50000

    load-server-state-from-file global

    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

listen stats
    bind *:8080 ssl crt /etc/haproxy/ssl/
    stats enable
    stats refresh 10s
    stats uri /
    stats show-legends
    stats show-node
    stats admin if stats_admin_ips
    stats http-request deny if !stats_access_ips
    http-request set-log-level silent

frontend external
    bind    :::80 v4v6
    bind    :::443 v4v6 ssl crt /etc/haproxy/ssl/ alpn h2,http/1.1
    redirect scheme https code 301 if !{ ssl_fc }
    option forwardfor
    http-request set-header X-Forwarded-Port  %[dst_port]
    http-request set-header X-Forwarded-Proto http  if !{ ssl_fc }
    http-request set-header X-Forwarded-Proto https if  { ssl_fc }

{{ $requiredTags := parseJSON `["live"]` }}

{{- range services -}}{{ if containsAll $requiredTags .Tags }}
    # Service {{ .Name }}
    acl {{ .Name }} hdr(host) {{ .Name }}.example.net
{{- end -}}{{- end }}

{{- range services -}}{{ if containsAll $requiredTags .Tags }}
    # Backend to use for {{ .Name }}
    use_backend {{ .Name }} if {{ .Name }}
{{- end -}}{{- end }}

{{ range services }}{{ if containsAll $requiredTags .Tags }}
{{ $service:=.Name }}
backend {{ .Name }}
{{- range service $service }}
{{- if .Tags | contains "live" }}
   server-template {{ .Node }} 1-5 live.{{ .Name }}.service.consul:{{ .Port }} resolvers consul resolve-opts allow-dup-ip resolve-prefer ipv4 check ssl verify none
{{- end -}}
{{- end }}{{ end }}{{ end }}

resolvers consul
    nameserver consul 127.0.0.1:8600
    accepted_payload_size 8192
    hold valid 5s

The consul-template process did build the configuration but here is what I can see in the daemon logs :

Sep 14 16:32:49 lb.example.net consul-template[129435]: 2022-09-14T16:32:49.557+0200 [WARN] (view) health.service(www|passing): Unexpected response code: 500 (rpc error: code = InvalidArgument desc = Key is required) (retry attempt 1 after "250ms")
Sep 14 16:34:25 lb.example.net consul-template[129435]: 2022-09-14T16:34:25.231+0200 [WARN] (view) health.service(www|passing): Unexpected response code: 500 (rpc error: code = InvalidArgument desc = Key is required) (retry attempt 1 after "250ms")

And these are the consul agent logs :

Sep 14 16:37:35 lb.example.net consul[19138]: 2022-09-14T16:37:35.132+0200 [ERROR] agent.rpcclient.health: subscribe call failed: err="rpc error: code = InvalidArgument desc = Key is required" failure_count=2160 key=www topic=ServiceHealth
Sep 14 16:37:35 lb.example.net consul[19138]: 2022-09-14T16:37:35.132+0200 [ERROR] agent.http: Request error: method=GET url=/v1/health/service/www?index=7028250&passing=1&stale=&wait=60000ms from=127.0.0.1:38948 error="rpc error: code = InvalidArgument desc = Key is required"

Do you guys have any idea why I’m seeing this? Did I miss something in my consul-template HAproxy template?

Thanks for your help.

2

Hey @drustan, thanks for asking.

It looks like this might have to do with the consul agent setting up a connection to the consul server via the streaming RPC API. Consul agent supports 2 methods for monitoring values in the consul servers. The first method is simple long polling which pulls the full data of the monitored value when an update occurs. The second is via a streaming RPC call that sends only the update-diff over a persistent stream. The second was added for performance on large clusters.

Some questions or things you could look at.

What version of consul?
Are the consul servers and consul agents the same version?
Are they both set up to use streaming?

Not 100% on that but it is my current best guess. Please ask any followup questions you might have and we can try to get to the bottom of this.

Thanks and good luck.

3

Thank you for your reply and the explanation about the existence of the two methods for monitoring values.

You asked me about the Consul version and I realized the agent was 1.13.1 and the servers were 1.12 so I upgraded and everything return to normal, it was that simple!

The error message wasn’t very clear but I sure should have checked that the versions matched first :-/

Have a great day!

4

Great news! I love easy fixes.

Thanks for using consul-template.

5

Unfortunately the problem still there, but it comes out when upgrading in multi DC configuration from 1.10.12 to 1.14.7.

agent.rpcclient.health: subscribe call failed: err=“rpc error: code = InvalidArgument desc = Key is required” failure_count=20 key=kube-system-apiserver topic=ServiceHealth

This log caused by DNS query from one updated DC to another DC with previous version.
Example:
From server node 1.14.7 make query like
consul-metrics.service.11012_NOT_UPDATED_DC.consul - failing

From server node 1.10.12 make query like
consul-metrics.service.1147_UPDATED_DC.consul - resolving

Update all consul servers in all clusters solving the issue, but it’s impossible to update all at once. I made it in test environment, but i don’t have an idea how to update my personal production environment without problems.