Hello,
I can’t connect an agent to a server in the consul cluster.
Log:
2-06-26T11:59:39.436+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:56688
2022-06-26T11:59:39.436+0300 [ERROR] agent.client.memberlist.lan: memberlist: failed to receive: Encryption is configured but remote state is not encrypted from=10.0.1.4:56688
2022-06-26T11:59:45.810+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:56710
2022-06-26T11:59:46.415+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:56714
2022-06-26T11:59:46.415+0300 [ERROR] agent.client.memberlist.lan: memberlist: failed to receive: Encryption is configured but remote state is not encrypted from=10.0.1.4:56714
2022-06-26T11:59:47.308+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: srv1-prod 10.0.1.4:8301
2022-06-26T11:59:53.115+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:56744
2022-06-26T11:59:53.115+0300 [ERROR] agent.client.memberlist.lan: memberlist: failed to receive: Encryption is configured but remote state is not encrypted from=10.0.1.4:56744
2022-06-26T11:59:58.429+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:56770
2022-06-26T11:59:58.430+0300 [ERROR] agent.client.memberlist.lan: memberlist: failed to receive: Encryption is configured but remote state is not encrypted from=10.0.1.4:56770
2022-06-26T12:00:00.422+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.0.1.4:8300 error="rpc error making call: i/o deadline reached"
I’ve checked that the encryption key is the same on the client machine and the server (in config and in keyring files).
I’ve tried:
- Remove local.keyring & remote.keyring.
- Create new token and use it as primary key.
- Update consul
If I add these line to configurations on all cluster members:
"encrypt_verify_incoming": false,
"encrypt_verify_outgoing": false
Then the log is changes:
2022-06-26T12:18:59.664+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:60270
2022-06-26T12:19:03.708+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:60286
2022-06-26T12:19:03.708+0300 [ERROR] agent.client.memberlist.lan: memberlist: Received invalid msgType (1) from=10.0.1.4:60286
2022-06-26T12:19:07.382+0300 [ERROR] agent.client: RPC failed to server: method=Coordinate.Update server=10.0.1.4:8300 error="rpc error making call: i/o deadline reached"
2022-06-26T12:19:07.382+0300 [ERROR] agent: Coordinate update error: error="rpc error making call: i/o deadline reached"
2022-06-26T12:19:08.922+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:60308
2022-06-26T12:19:08.922+0300 [ERROR] agent.client.memberlist.lan: memberlist: Received invalid msgType (1) from=10.0.1.4:60308
2022-06-26T12:19:14.869+0300 [DEBUG] agent.client.memberlist.lan: memberlist: Stream connection from=10.0.1.4:60330
2022-06-26T12:19:14.869+0300 [ERROR] agent.client.memberlist.lan: memberlist: Received invalid msgType (1) from=10.0.1.4:60330
2022-06-26T12:19:14.909+0300 [ERROR] agent.client: RPC failed to server: method=Catalog.NodeServiceList server=10.0.1.4:8300 error="rpc error making call: i/o deadline reached"
2022-06-26T12:19:14.909+0300 [ERROR] agent.anti_entropy: failed to sync remote state: error="rpc error making call: i/o deadline reached"
How can solve this issue?
UPD: I’ve removed all files in Consul dir (/opt/consul using RHEL) and errors have been disappered. But what reasons of the problem& In further I have to know what I need to delete exactly.