gRPC error in server logs without enabling gRPC

Consul
1

Hi,

I can see the following error in my Consul server (v1.9.4) logs, related to a connection coming from consul clients (v1.10.4) -

[ERROR] agent: gRPC conn opened but gRPC RPC is disabled, closing: conn=from=<Client_IP>:50408

At the same time, using tcpdump, I can see this client attempting communication with the server on port 8300.

I did not enable any grpc option in neither of the configuration files, and I understand that by default it’s disabled, so what could be the issue?

Thanks

3

Hi @Talmazor,

We recommend upgrading Consul servers to a newer version before upgrading clients. This is called out on consul.io: Upgrading Consul - Standard Upgrades.

Consul 1.10 introduced a new RPC transport based on gRPC called streaming. It is enabled by default for 1.10 clients and servers. Here’s the excerpt on it from the 1.10 release highlights.

  • Streaming Enabled by Default for Service Health: Streaming is a major architectural enhancement in how update notifications for blocking queries are delivered within the cluster which significantly reduces CPU and network bandwidth usage for large-scale Consul deployments. In Consul 1.10, streaming is now available for the service health HTTP endpoint and is enabled by default.

You are receiving this error because the Consul 1.10 client agents are not able to initiate a gRPC connection over port 8300 to the Consul 1.9 servers.

You can resolve this through one of two methods.

  1. Upgrade the Consul servers to 1.10 in order to allow the agents to utilize the new streaming RPC transport.

  2. Disable streaming on the 1.10 client agents by setting use_streaming_backend to false.

    # agent-config.hcl
use_streaming_backend = false

I hope this helps. Let me know if you have any questions.

1 Like