Enabling TLS in Prod Environment

Vault
1

I have a two-node Vault setup running on Windows EC2 instances, fronted by an ELB with TLS termination at the load balancer level. My current configuration has TLS disabled for Vault itself.

My current configuration for each server looks like this:

server1

api_addr = "http://${PRIVATE_IP}:8200"
cluster_addr = "http://${PRIVATE_IP}:8201"

listener "tcp" {
	address     = "0.0.0.0:8200"
	tls_disable = 1
	telemetry {
		unauthenticated_metrics_access = true
	}
}


storage "dynamodb" {
	ha_enabled = "true"
	region     = "us-east-1"
	table      = "{tableid}"
}
ui = true

server2

api_addr = "http://${PRIVATE_IP}:8200"
cluster_addr = "http://${PRIVATE_IP}:8201"

listener "tcp" {
	address     = "0.0.0.0:8200"
	tls_disable = 1
	telemetry {
		unauthenticated_metrics_access = true
	}
}

storage "dynamodb" {
	ha_enabled = "true"
	region     = "us-east-1"
	table      = "{tableid}"
}
ui = true

Vault_ADDR is currently set to “127.0.0.1:8200”

I want to enable TLS for internal communication between the Vault servers and between the servers and the ELB. I modified the configuration below for each server using same certificate :

listener "tcp" {
  address     = "0.0.0.0:8200"
  tls_cert_file = "C:\\vault\\vault_tls.crt"
  tls_key_file  = "C:\\vault\\vault_tls.key"
  telemetry {
    unauthenticated_metrics_access = true
  }
}

I am no expert when it comes to adding certs for TLS, So I am using wildcard certificate from Let’s Encrypt, attached to my ELB (*.myvaultsite.com).

when I apply this certificate to Vault, I get TLS handshake errors:

TLS handshake error from 127.0.0.1:51402: remote error: tls: bad certificate

Is it possible to use the same public wildcard certificate I use in ELB for internal communication between Vault servers since I want to avoid creating self signed certificate. What changes do I need to make in my Vault setup to fix this TLS handshake error.

Thanks for your help!

2

any recommendation on this?

3

Hi lucardcoder,

I’m a vault newbie, but I’ve a lot of experience in certificate management and setting up webservers and load balancer TLS. I’ve also successfully deployed a simple non load balanced vault internally which is using TLS, so I may be able to help you.

Check the common name and/or SAN details in the certificate located at C:\vault\vault_tls.crt are. Those are the IPs/FQDN you should access the vault servers using. If you use an IP or FQDN which isn’t in the certificate it is expected you’d get something akin to an certificate mismatch error in a Web Browser (I’d say tls: bad certificate is the vault clients equivalent to that).

If the certificate at C:\vault\vault_tls.crt has a common name and/or SAN of 127.0.0.1 then new certificates need generating with proper DNS / IPs for the relevant servers in.

If the certificate at C:\vault\vault_tls.crt has a common name and/or SAN of *.myvaultsite.com it may just a case of making sure you set VAULT_ADDR to server1.myvaultsite.com on server1 and server2.myvaultsite.com on server2.

I’m not familiar with ELB so I can’t give you an specific advice, but I can give you some general advice… I can imagine a situation where ELB is configured to load balance myvault.myvaultsite.com between server1 and server2 ?

I would need to make sure the ELB is configured to use IP(s) or FQDN(s) which is present in the certificate C:\vault\vault_tls.crt.

I would also need to make sure that ELB trusts the CA which signed the certificates used by server1 and 2 by doing one of the following…

  • Signed by a Public CA (such as LetsEncrypt) and that the ELB trusts that CA
    or
  • Signed by an internal CA (Microsoft Active Directory Certificate Service) and that the ELB is configured to trust that internal CA.

Remember TLS performs two important jobs. Encryption and verification.

If the IP/FQDN you use to access vault isn’t in the certificate you’ll get something akin to a certificate mistmatch error.
If the certificate is signed by an untrusted CA (because it’s internally signed, self signed or signed by a public CA ELB doesn’t trust) then you’ll get something akin to a untrusted certificate error.

Good luck!

Regards

Steve

1 Like
4

Thank you for your prompt and detailed response @SteveScotter. I will try your suggestions and report back.

1 Like