Unable to access Vault UI from the LoadBalancer

vempasiddhartha · January 26, 2023, 1:10pm

I am facing issue w.r.t to accessing the vault ui. I provisioned vault using helm chart with TLS and UI enabled in private EKS cluster in private subnets which has NAT Gateway in route table. A classic load balancer got created since I enabled UI in the helm vaules.yaml file. However, when I start using the load balancer url in the browser I keep seeing a message “Client sent an HTTP request to an HTTPS server.” In a similar way, I also provisioned Grafana using helm chart. Initially the grafana service got created with Cluster IP. Later I edited the service type to Load Balancer using the kubectl edit svc command. Then a new load balancer got provisioned and I am able to open the Grafana login page using the Loadbalancer url. But with vault it is not happening. Can someone please tell me what could be the issue here with my vault? Giving the values.yaml file here.

cat > ${WORKDIR}/overrides.yaml <<EOF
global:
enabled: true
tlsDisable: false
injector:
enabled: true
image:
repository: “hashicorp/vault-k8s”
tag: “latest”
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m
server:
readinessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204”
livenessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true”
initialDelaySeconds: 60
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
volumes:
- name: userconfig-vault-ha-tls
secret:
defaultMode: 420
secretName: vault-ha-tls
volumeMounts:
- mountPath: /vault/userconfig/vault-ha-tls
name: userconfig-vault-ha-tls
readOnly: true
auditStorage:
enabled: false
standalone:
enabled: false
affinity: “”
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener “tcp” {
tls_disable = 0
address = “[::]:8200”
cluster_address = “[::]:8201”
tls_cert_file = “/vault/userconfig/vault-ha-tls/vault.crt”
tls_key_file = “/vault/userconfig/vault-ha-tls/vault.key”
tls_client_ca_file = “/vault/userconfig/vault-ha-tls/vault.ca”
}
storage “raft” {
path = “/vault/data”
}
disable_mlock = true
service_registration “kubernetes” {}
nodeSelector:
nodeName: vault
ui:
enabled: true
serviceType: “LoadBalancer”
serviceNodePort: null
externalPort: 8200

EOF

When I use https for invoking the url either the load balancer url or the local host i am getting the below issue.

when i invoke the vault service url with curl and https by passing the -k flag then i am getting correct response.
curl -k https://127.0.0.1:8200/v1/sys/health

stuart-c · January 26, 2023, 4:12pm

Are you specifically using a classic load balancer instead of an ALB or NLB? If so, are you using it in layer 4 or layer 7 mode?

If layer 7 mode it suggests you need to ensure the configuration for the loadbalancer backend is set to HTTPS instead of HTTP.

If layer 4 you need to ensure you are using https:// in your browser.

vempasiddhartha · January 26, 2023, 6:02pm

Hi Stuart-c! Once again thank you for stepping in and trying to help me. Below are my answers to your questions.
Are you specifically using a classic load balancer instead of an ALB or NLB? → I haven’t explicitly specified to use classic loadbalancer. In the vault helm chart values.yaml file I have used the below values and so the chart itself created a vault service of the type loadbalancer which has provisioned an classic loadbalancer.
ui:
enabled: true
serviceType: “LoadBalancer”
serviceNodePort: null
externalPort: 8200

If so, are you using it in layer 4 or layer 7 mode? → The helm provisioned classic load balancer is configured in layer 4 mode.

If layer 4 you need to ensure you are using https:// in your browser. → Yes, Stuart, I am using https and then appending the load balancer url. In that case I am getting the below error.

When i click on the Advanced Option then I am getting the below page.

When I click on the proceed to option present at the last of the page, then i am getting the below popup window. I understand it is due to network setup jamf.

When I just cancel it selecting any options listed in the popup, then i am getting the vault login page but without https. I don’t want that. I want the https enabled. Can you please advise how to enable https in the browser as well. I know that I enabled https in the vault values.yaml file but that is not helping when trying to reach the UI from browser.

stuart-c · January 26, 2023, 6:19pm

That bottom image isn’t showing you not using HTTPS. You are in fact using HTTPS.

However it is saying the certificate is untrusted, which is why you are getting the initial error and “not secure” warning. There are two options to stop getting the error/warning:

Add the CA to your browser so it is now trusted
Change the certificate in Vault to one that is trusted by your browser

Option 1 is good if you only need a single browser to be updated, or if in a corporate setting it is easy to “push” browser settings to everyone’s computer.

Option 2 generally requires you to purchase a certificate or setup something like Lets Encrypt, so it a bit more effort and/or money, but means every browser will work without trust errors. You would also need to ensure you are using a Route53 zone and your own DNS name instead of the automatically produced AWS loadbalancer DNS name.

vempasiddhartha · January 27, 2023, 2:20am

Hi Stuart! The setup I did is for production environment. So I have two requirements.

All the communication between vault and its clients(pods) should be secured. So I enabled TLS while setting up the cluster.
I also want the traffic between browser and vault-ui should be secured.

In order to make the communication between vault and its clients secure, I followed the below documentation.

Can you please let me know if this way of certificate generation works fine for production environment? If Yes, then no issues.
If No, can you please let me know if I need to get it generated from authorized CA’s like Lets Encrypt? My doubt is how would authorized CA’s generate certificates for the domains(like DNS.1 = *.vault
DNS.2 = vault.vault.svc
DNS.3 = vault.vault.svc.cluster.local
DNS.4 = *.vault-internal
DNS.5 = *.vault-internal.svc.cluster.local
IP.1 = 127.0.0.1) that are internal to an EKS cluster?

When I use the above given documentation, three files got generated which I used in my helm values.yaml file. They are vault.ca, vault.crt, vault.key
So far I am in an impression that vault would use these certificate related files only to make vault-client communication secure. But, by seeing the errors in the browser I feel the same are being used even for browser - vault-ui communication as well. Please correct me if I am wrong. Please help me to sort out this issue.

vempasiddhartha · January 28, 2023, 10:41am

@stuart-c Can you please help on this? I waiting for your response.

stuart-c · January 28, 2023, 1:07pm

As you mention you can’t use any publicly trusted CA (such as Lets Encrypt or other paid CAs) if you are wanting to include non public DNS names (such as within cluster names). The only option for your Vault certificate is therefore your own internal CA.

If you are wanting to remove warnings on your browser you can import that private CA and mark it as trusted. You could also look at using multiple listener blocks in your Vault configuration (using different ports) to try to have separate certificates for the within cluster & external users - with the external users being presented with a certificate from Lets Encrypt or a paid provider. Another option (although generally not seen as good practice) is to switch to an ALB so that external HTTPS traffic terminates at the load balancer (using a publicly trusted certificate).

alain · January 29, 2023, 3:13pm

First - I would recommend doing a deeper dive in TLS, certificates and trust stores, and understand the errors you are getting. The fundamental error you are getting is that the Browser does not trust the Certifying Authority ( CA ) of xxxxx.elb.amazonaws.com you are using.

For the Web UI - if you are going on the public internet ( an assumption as you are using the public name of your load balancer ), you are going to need a DNS name. but you probably figured that out.

For the certificate - if you do no use a Public CA to sign your certificates, you will need to add the CA to the truststore of your browser ( you can google that ). note that this is meant to protect the CLIENT from a Man In The Middle attack ( Again reference back to the TLS deeper dive - I found Bulletproof TLS Guide while looking at the let’s encrypt site.)

As for internal / external - I have not setup vault as you are doing but have had issues with different destination IP’s and the names. One items I have used splitdns (and even separate DNS servers for internal and external) - basically one IP is returned internally, and another is returned externally. This may not be what you are looking for, but the following article is similar to what you want : Split Horizon DNS with external-dns and cert-manager for Kubernetes

Yes they are - there is no separation between the API and UI endpoint. That also means, if you expose the UI to the outside - YOU ARE EXPOSING THE API. The UI ( in the browser ) uses JavaScript to make API requests to vault - if you block the API, the UI will not work.

One strategy would be to have a limited UI for only what you need - and something internal that does the actual communication to Vault. Then your TLS issue also becomes simpler as there is separation between the Exposed UI and Vault.

vempasiddhartha · February 16, 2023, 10:43am

@stuart-c I am very much thankful to you for the support. I have finally achieved my requirement using the below configuration in the values.yaml
global:
enabled: true
replicas: 1
tlsDisable: false
injector:
enabled: true
image:
repository: “hashicorp/vault-k8s”
tag: “latest”
authPath: “auth/kubernetes”
logLevel: “debug”
logFormat: “standard”
resources:
requests:
memory: 256Mi
cpu: 250m
limits:
memory: 256Mi
cpu: 250m
server:
logLevel: “debug”
logFormat: “standard”
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: “role-arn-to-access-kms”
readinessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204”
livenessProbe:
enabled: true
path: “/v1/sys/health?standbyok=true”
initialDelaySeconds: 60
extraEnvironmentVars:
VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
AWS_ROLE_SESSION_NAME: vault_access_to_aws
volumes:
- name: userconfig-vault-ha-tls
secret:
defaultMode: 420
secretName: vault-ha-tls
volumeMounts:
- mountPath: /vault/userconfig/vault-ha-tls
name: userconfig-vault-ha-tls
readOnly: true
auditStorage:
enabled: false
standalone:
enabled: false
affinity: “”
nodeSelector:
nodeName: vault
ha:
enabled: true
replicas: 3
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener “tcp” {
tls_disable = 0
address = “[::]:8200”
cluster_address = “[::]:8201”
tls_cert_file = “/vault/userconfig/vault-ha-tls/vault.crt”
tls_key_file = “/vault/userconfig/vault-ha-tls/vault.key”
tls_client_ca_file = “/vault/userconfig/vault-ha-tls/vault.ca”
}
storage “raft” {
path = “/vault/data”
retry_join {
leader_api_addr = “https://vault-0.vault-internal:8200”
leader_ca_cert_file = “/vault/userconfig/vault-ha-tls/vault.ca”
leader_client_cert_file = “/vault/userconfig/vault-ha-tls/vault.crt”
leader_client_key_file = “/vault/userconfig/vault-ha-tls/vault.key”
}
retry_join {
leader_api_addr = “https://vault-1.vault-internal:8200”
leader_ca_cert_file = “/vault/userconfig/vault-ha-tls/vault.ca”
leader_client_cert_file = “/vault/userconfig/vault-ha-tls/vault.crt”
leader_client_key_file = “/vault/userconfig/vault-ha-tls/vault.key”
}
retry_join {
leader_api_addr = “https://vault-2.vault-internal:8200”
leader_ca_cert_file = “/vault/userconfig/vault-ha-tls/vault.ca”
leader_client_cert_file = “/vault/userconfig/vault-ha-tls/vault.crt”
leader_client_key_file = “/vault/userconfig/vault-ha-tls/vault.key”
}

              autopilot {
                 cleanup_dead_servers = "true"
                 last_contact_threshold = "200ms"
                 last_contact_failure_threshold = "10m"
                 max_trailing_logs = 250000
                 min_quorum = 2
                 server_stabilization_time = "10s"
            }
        }

        disable_mlock = true
        service_registration "kubernetes" {}

        seal "awskms" {
              region     = "region"
              kms_key_id = "kms_key_id"
            }

ui:
enabled: true
serviceType: “LoadBalancer”
serviceNodePort: null
externalPort: 443
annotations:
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: “cert-arn”
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: https
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: “443,8200”
service.beta.kubernetes.io/do-loadbalancer-healthcheck-path: “/ui/”
service.beta.kubernetes.io/aws-load-balancer-internal: “false”

vempasiddhartha · February 16, 2023, 10:45am

@stuart-c I want to know if there is a way I can pass my vault secrets into the pod container as an environment variable. I have searched for it in various sites but i couldn’t. Can you please help me on this?

hardsingh · November 1, 2023, 3:48pm

If you are using Helm to deploy vault, you can add the env variables in the values.yaml file using the following stanza. Ensure you have created the secret beforehand. In my case I was passing the root token as a secret.

extraSecretEnvironmentVars:
      - envName: VAULT_TOKEN
        secretName: vault-token
        secretKey: VAULT_TOKEN

Topic		Replies	Views
Vault UI 404 - Helm chart in EKS Vault	0	392	May 24, 2021
Https not working for vault nlb Vault vault	1	164	May 29, 2024
Not able to access vault UI from browser- Gateway timeout Vault k8s , vault	0	260	October 12, 2023
Vault UI Loadbalancer created as internal Vault vault	1	97	May 23, 2024
How to expose vault HTTPS API that installed in k8s to outside clients Vault k8s , connect , vault	13	1786	January 25, 2021

Unable to access Vault UI from the LoadBalancer

Related topics