Hello!
I have a new Nomad server C which I want it to join an existing Nomad cluster of 2 servers A and B.
nomad agent -config /etc/nomad.d/nomad.hcl -config /etc/nomad.d/server.hcl
However, I get the error “failed to receive: No installed keys could decrypt the message” on the server A when I start the new server C.
This could be either mTLS which is enabled (and working fine) on the 2 servers (A and B) from the cluster, or it could also be the gossip protocol encryption.
The certificate for mTLS was generated using
nomad tls cert create -ca path-to-ca.pem -key path-to-ca-key.pem -server -additional-ipaddress "192.168.1.48" -additional-dnsname "nugget.global.nomad"
and was then moved to the new server C.
(Note that the IP address in the certificate do not correspond to the IP address of the new host, but this is not a problem since the server B has the same “mistake” and it works just fine.)
The gossip encryption key is the same (I used vimdiff between the 3 server config files to make sure there was no typo), which does not match with the only other thread I found online about this error.
I tried running both servers A and C with -log-level TRACE but didn’t get any more log.
I tried to join A with it being either leader or follower with no luck.
I even tried to join B instead of A, still the same error.
You can find the two config files for the new server C below, as well as both logs from A and C when trying to join the cluster.
I hope someone can help or at least, point me in the right direction, since the logs are so poor.
Best regards,
Virgile.
C - nomad.hcl
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: BUSL-1.1
# Full configuration options can be found at https://developer.hashicorp.com/nomad/docs/configuration
data_dir = "/opt/nomad/data"
bind_addr = "0.0.0.0"
datacenter = "paris"
enable_syslog = true
tls {
http = true
rpc = true
ca_file = "/etc/nomad.d/security/nomad-agent-ca.pem"
verify_server_hostname = true
verify_https_client = true
}
acl {
enabled = true
}
C - server.hcl
name = "nugget"
server {
enabled = true
bootstrap_expect = 3
encrypt = "********************************************"
raft_boltdb {
no_freelist_sync = true
}
raft_multiplier = 10
server_join {
retry_join = ["192.168.1.21:4648"]
}
job_gc_threshold = "168h"
eval_gc_threshold = "168h"
batch_eval_gc_threshold = "168h"
deployment_gc_threshold = "168h"
}
tls {
cert_file = "/etc/nomad.d/security/global-nugget-nomad.pem"
key_file = "/etc/nomad.d/security/global-nugget-nomad-key.pem"
}
C - logs
2025-09-24T13:33:39.158+0200 [INFO] agent.joiner: starting retry join: agent_mode=server servers=192.168.1.21:4648
2025-09-24T13:33:39.160+0200 [DEBUG] nomad: memberlist: Initiating push/pull sync with: 192.168.1.21:4648
2025-09-24T13:33:39.163+0200 [DEBUG] nomad: memberlist: Failed to join 192.168.1.21:4648: No installed keys could decrypt the message
2025-09-24T13:33:39.163+0200 [WARN] agent.joiner: join failed: agent_mode=server
error=
| 1 error occurred:
| \t* Failed to join 192.168.1.21:4648: No installed keys could decrypt the message
|
retry=30s
A - logs
2025-09-24T13:33:39.222+0200 [DEBUG] nomad: memberlist: Stream connection from=192.168.1.133:45006
2025-09-24T13:33:39.223+0200 [ERROR] nomad: memberlist: failed to receive: No installed keys could decrypt the message from=192.168.1.133:45006