Failed to join No installed keys could decrypt the message

Consul
1

Hello. I can’t run consul on clients. (on slaves).
Server:

{
 "bootstrap": true,
 "server": true,
 "datacenter": "dc1",
 "data_dir": "/opt/consul",
 "encrypt": "OPmHjiduFjb5R3uiJsjZAgsU+EoJr1ZdirFFfxM4lto=",
 "log_level": "INFO",
 "enable_syslog": true,
 "start_join": ["172.18.0.3"],
 "advertise_addr": "172.18.0.3"
}

Client:

{
    "datacenter": "dc1",
    "data_dir": "/opt/consul",
    "log_level": "DEBUG",
    "node_name": "test2.local",
    "server": false,
    "encrypt": "OPmHjiduFjb5R3uiJsjZAgsU+EoJr1ZdirFFfxM4lto=",
    "addresses": {
      "dns": "127.0.0.1",
      "https": "0.0.0.0",
      "grpc": "127.0.0.1",
      "http": "127.0.0.1"
    },
    "bind_addr": "172.18.0.2",
    "start_join": ["172.18.0.3"],
    "ports": {
      "dns": 53
     },
    "advertise_addr": "172.18.0.2"
}

Encrypt keys are the same but I get an error trying to launch the consul on the client:

Failed to join 172.18.0.3:8301: No installed keys could decrypt the message

I also tried to remove encrypt line in configs, but I get the same error.

1 Like
2

Fixed. I removed the local.keyring file on the /opt/consul/serf in all nodes.

1 Like
3

Hi @wusikijeronii,

Did you by chance attempt to change the encryption key at some point? If so, the process for changing the encryption is slightly more involved.

When a Consul agent starts for the first time, it will read and attempt to use the encryption key defined in the encrypt parameter of the agent’s configuration file. After the initial startup, the agent will create and store the gossip key in the $DATA_DIR/consul/serf/local.keyring file.

At this point, if the agent is subsequently restarted, the gossip key will be read from the keyring file, not the encrypt parameter in the agent configuration. (This behavior can be changed by setting disable_keyring_file to true in the agent config.)

If you need to change/rotate the key, you can do so using the consul keyring command. This command allows for you to install a new key, and safely rotate all agents to the key without requiring each individual agent’s configuration be updated, or each agent be restarted for the new key to take effect.

A step-by-step tutorial for rotating gossip keys can be found here.

2 Likes
4

Good morning.
Yes, I did. But I also have tried to change the encrypt key. In my case the reason of the issue is a duplicating keys. So, I’ve ran keyring command on each node. Seems when I launched this one on the second node the encrypt key was been created and moved to local keyring file. It is both a cause and a solution 'cos created key on the second node differed from key in the first node.

1 Like
5

thank you @blake

you added a couple hours to my life expectancy