I am using Vault 1.6.0 with raft and have Cloud (AWS) auto_join working and nodes talking to each other over HTTPS, but I’m wondering why the requirement to embed the IP address in the subjectAltName (SAN). It makes it virtually impossible to use with DHCP and/or auto-scaling groups.
Requiring that the SSL cert have the IP address embedded seems at odds with how the public Internet works, where the Certificate Authority vouches for the authenticity of the cert. Each node is now a “pet” instead of “cattle”, to borrow that analogy, because the node has to be built and its IP known (and it cannot change) before creating and signing the CSR.
Is there some way to turn off the IP SAN requirement? Does it support wildcards? I know about -tls-skip-verify for CLI commands, but that doesn’t seem to help with the server itself.
I’m confused. I didn’t think that was a requirement.
I know you can include 127.0.0.1 in some infrastructures, to enable CLI and cURL usage of certificates. But, as you say, anything beyond that is very limiting.
Can you provide more information about what’s failing in your infrastructure, before you took the step of including IP SANs?
Vault itself seems to be requiring the IP SAN entries. 127.0.0.1 alone isn’t enough.
An indicator of Vault’s reliance on the IP SAN field is shown in this error message (I changed the IP address of the node, causing the error): core: join attempt failed: error="error during raft bootstrap init call: Put "https://10.0.11.68:8200/v1/sys/storage/raft/bootstrap/challenge": x509: certificate is valid for 10.0.11.63, 127.0.0.1, not 10.0.11.68"
No, that’s correct: not a requirement. I’m just getting up to speed, mind; no expert. But I have a Consul-backed cluster and a Raft-backed one, both communicating over TLS exclusively – one even auto-seals the other via transit! – and I haven’t had to worry about IP addresses at all.
With only 127.0.0.1 as the IP SAN entry, this is the error message:
Dec 4 17:22:51 ip-10-0-11-68 vault[54576]: 2020-12-04T17:22:51.591Z [WARN] core: join attempt failed: error="error during raft bootstrap init call: Put "https://10.0.85.90:8200/v1/sys/storage/raft/bootstrap/challenge": x509: certificate is valid for 127.0.0.1, not 10.0.85.90"
At this point it looks to me like IP SAN is required by Vault. That’s a non-Cloud approach, in my view, and I’d love to know how to work around or fix it.
I’m confused here.
Are you trying to use https://10.0.11.68:8200 and expect a cert for vault.yourco.com will work?
Vault should be served up and consumed by clients with a domain name, not the IP.
This is the automatic discovery feature in 1.6. As the logs above show, the hosts themselves are querying the environment for specific tags that indicate a potential Vault peer.
If Vault requires a DNS name, then I would expect Vault to attempt a DNS lookup based on the IP that the Vault discovery mechanism found.
@mikegreen’s comment had an interesting clue. Using the DNS name and manually joining the node seems to work. However, this completely defeats the purpose of Cloud auto_join.
Right now this looks like a design flaw in the discover-aws module. It scans and finds the other EC2 instances, but tries to connect via IP address, which leads to the IP SAN rabbit hole.
It seems like a simple solution would be to perform a reverse lookup on the IP address before connecting.