I am trying to run a Vault cluster of 5 pods using the official Helm chart. I can provide more details if necessary, but I think at this stage the problem I am experiencing is not Kubernetes-specific.
I don’t want to use TLS, so I disabled/removed all TLS-related configuration options. Looking at the pods’ logs, it seems they can’t manage to talk to each other. All of them report the same messages in their logs:
[INFO] core: attempting to join possible raft leader node: leader_addr=http://vault-2:8200
[WARN] core: join attempt failed: error="error during raft bootstrap init call: Put \"http://vault-2:8200/v1/sys/storage/raft/bootstrap/challenge\": dial tcp: lookup vault-2 on 10.96.0.10:53: server misbehaving"
I did some research on this “server misbehaving” error message, but couldn’t find anything on the internet. I suspect that pods expect other pods to talk HTTPS, and aren’t happy when they get a plain HTTP response. Could that be the cause of this “server misbehaving” message?
Thanks for any help! If you require more details (version numbers, configuration, logs, etc.) please ask and I will provide them.
No RAFT protocol does not require TLS.
I’ll start by saying I’m a kubernetes noob and most likely moving to Nomad for my use cases anyway. That said, sounds like the most likely issue with that message is some garbled data from kubs dns-system. Are you using external-names or service registration? If not you need to statically call out the names of the other nodes in your configuration – most likely is that the names you are using are not getting translated to the right node.