I am facing issue w.r.t to accessing the vault ui. I provisioned vault using helm chart with TLS and UI enabled in private EKS cluster in private subnets which has NAT Gateway in route table. A classic load balancer got created since I enabled UI in the helm vaules.yaml file. However, when I start using the load balancer url in the browser I keep seeing a message “Client sent an HTTP request to an HTTPS server.” In a similar way, I also provisioned Grafana using helm chart. Initially the grafana service got created with Cluster IP. Later I edited the service type to Load Balancer using the kubectl edit svc command. Then a new load balancer got provisioned and I am able to open the Grafana login page using the Loadbalancer url. But with vault it is not happening. Can someone please tell me what could be the issue here with my vault? Giving the values.yaml file here.
Hi Stuart-c! Once again thank you for stepping in and trying to help me. Below are my answers to your questions.
Are you specifically using a classic load balancer instead of an ALB or NLB? → I haven’t explicitly specified to use classic loadbalancer. In the vault helm chart values.yaml file I have used the below values and so the chart itself created a vault service of the type loadbalancer which has provisioned an classic loadbalancer.
If so, are you using it in layer 4 or layer 7 mode? → The helm provisioned classic load balancer is configured in layer 4 mode.
If layer 4 you need to ensure you are using https:// in your browser. → Yes, Stuart, I am using https and then appending the load balancer url. In that case I am getting the below error.
When I just cancel it selecting any options listed in the popup, then i am getting the vault login page but without https. I don’t want that. I want the https enabled. Can you please advise how to enable https in the browser as well. I know that I enabled https in the vault values.yaml file but that is not helping when trying to reach the UI from browser.
That bottom image isn’t showing you not using HTTPS. You are in fact using HTTPS.
However it is saying the certificate is untrusted, which is why you are getting the initial error and “not secure” warning. There are two options to stop getting the error/warning:
Add the CA to your browser so it is now trusted
Change the certificate in Vault to one that is trusted by your browser
Option 1 is good if you only need a single browser to be updated, or if in a corporate setting it is easy to “push” browser settings to everyone’s computer.
Option 2 generally requires you to purchase a certificate or setup something like Lets Encrypt, so it a bit more effort and/or money, but means every browser will work without trust errors. You would also need to ensure you are using a Route53 zone and your own DNS name instead of the automatically produced AWS loadbalancer DNS name.
Hi Stuart! The setup I did is for production environment. So I have two requirements.
All the communication between vault and its clients(pods) should be secured. So I enabled TLS while setting up the cluster.
I also want the traffic between browser and vault-ui should be secured.
In order to make the communication between vault and its clients secure, I followed the below documentation.
Can you please let me know if this way of certificate generation works fine for production environment? If Yes, then no issues.
If No, can you please let me know if I need to get it generated from authorized CA’s like Lets Encrypt? My doubt is how would authorized CA’s generate certificates for the domains(like DNS.1 = *.vault
DNS.2 = vault.vault.svc
DNS.3 = vault.vault.svc.cluster.local
DNS.4 = *.vault-internal
DNS.5 = *.vault-internal.svc.cluster.local
IP.1 = 127.0.0.1) that are internal to an EKS cluster?
When I use the above given documentation, three files got generated which I used in my helm values.yaml file. They are vault.ca, vault.crt, vault.key
So far I am in an impression that vault would use these certificate related files only to make vault-client communication secure. But, by seeing the errors in the browser I feel the same are being used even for browser - vault-ui communication as well. Please correct me if I am wrong. Please help me to sort out this issue.
As you mention you can’t use any publicly trusted CA (such as Lets Encrypt or other paid CAs) if you are wanting to include non public DNS names (such as within cluster names). The only option for your Vault certificate is therefore your own internal CA.
If you are wanting to remove warnings on your browser you can import that private CA and mark it as trusted. You could also look at using multiple listener blocks in your Vault configuration (using different ports) to try to have separate certificates for the within cluster & external users - with the external users being presented with a certificate from Lets Encrypt or a paid provider. Another option (although generally not seen as good practice) is to switch to an ALB so that external HTTPS traffic terminates at the load balancer (using a publicly trusted certificate).
First - I would recommend doing a deeper dive in TLS, certificates and trust stores, and understand the errors you are getting. The fundamental error you are getting is that the Browser does not trust the Certifying Authority ( CA ) of xxxxx.elb.amazonaws.com you are using.
For the Web UI - if you are going on the public internet ( an assumption as you are using the public name of your load balancer ), you are going to need a DNS name. but you probably figured that out.
For the certificate - if you do no use a Public CA to sign your certificates, you will need to add the CA to the truststore of your browser ( you can google that ). note that this is meant to protect the CLIENT from a Man In The Middle attack ( Again reference back to the TLS deeper dive - I found Bulletproof TLS Guide while looking at the let’s encrypt site.)
As for internal / external - I have not setup vault as you are doing but have had issues with different destination IP’s and the names. One items I have used splitdns (and even separate DNS servers for internal and external) - basically one IP is returned internally, and another is returned externally. This may not be what you are looking for, but the following article is similar to what you want : Split Horizon DNS with external-dns and cert-manager for Kubernetes – Techno Wizardry
One strategy would be to have a limited UI for only what you need - and something internal that does the actual communication to Vault. Then your TLS issue also becomes simpler as there is separation between the Exposed UI and Vault.
@stuart-c I want to know if there is a way I can pass my vault secrets into the pod container as an environment variable. I have searched for it in various sites but i couldn’t. Can you please help me on this?