Access vault web UI with TLS enabed + raft

Vault
,
1

Hello

I have installed vault in my cluster with this article: Vault Installation to Minikube via Helm with TLS enabled

Here are my values:

global:
  enabled: true
  tlsDisable: false

injector:
  enabled: true
server:
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-ha-tls/vault.ca
    VAULT_TLSCERT: /vault/userconfig/vault-ha-tls/vault.crt
    VAULT_TLSKEY: /vault/userconfig/vault-ha-tls/vault.key
  volumes:
  - name: userconfig-vault-ha-tls
    secret:
      defaultMode: 420
      secretName: vault-ha-tls
  volumeMounts:
  - mountPath: /vault/userconfig/vault-ha-tls
    name: userconfig-vault-ha-tls
    readOnly: true
  
  ingress:
    enabled: true
    annotations:
      cert-manager.io/cluster-issuer: letsencrypt-prod
    ingressClassName: "nginx"
    pathType: Prefix
    # When HA mode is enabled and K8s service registration is being used,
    # configure the ingress to point to the Vault active service.
    activeService: true
    hosts:
    - host: MYDOMAIN.COM
      paths: []
    tls:
    - secretName: MYDOMAIN.COM
      hosts:
        - MYDOMAIN.COM
  
  dataStorage:
    enabled: true
    # Size of the PVC created
    size: 10Gi
    # Location where the PVC will be mounted.
    mountPath: "/vault/data"
    storageClass: openebs-cstor
    accessMode: ReadWriteOnce
  
  dev:
    enabled: false
  standalone:
    enabled: false
  affinity: ""
  ha:
    enabled: true
    replicas: 3
    # Enables Raft integrated storage
    raft:
      enabled: true
      setNodeId: true
      config: |
        ui = true

        listener "tcp" {
          tls_disable = 0
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-ha-tls/vault.crt"
          tls_key_file  = "/vault/userconfig/vault-ha-tls/vault.key"
          tls_client_ca_file = "/vault/userconfig/vault-ha-tls/vault.ca"
          # Enable unauthenticated metrics access (necessary for Prometheus Operator)
          telemetry {
            unauthenticated_metrics_access = "true"
          }
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}

ui:
  enabled: true

Everything seems is OK!

$ k -n vault get all
NAME                                       READY   STATUS    RESTARTS   AGE
pod/vault-0                                1/1     Running   0          3h43m
pod/vault-1                                1/1     Running   0          3h43m
pod/vault-2                                1/1     Running   0          3h43m
pod/vault-agent-injector-fc98d66b5-f5s8m   1/1     Running   0          3h43m

NAME                               TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/vault                      ClusterIP   10.110.65.221    <none>        8200/TCP,8201/TCP   3h43m
service/vault-active               ClusterIP   10.110.12.120    <none>        8200/TCP,8201/TCP   3h43m
service/vault-agent-injector-svc   ClusterIP   10.102.65.117    <none>        443/TCP             3h43m
service/vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   3h43m
service/vault-standby              ClusterIP   10.111.223.155   <none>        8200/TCP,8201/TCP   3h43m
service/vault-ui                   ClusterIP   10.104.148.124   <none>        8200/TCP            3h43m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-agent-injector   1/1     1            1           3h43m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-agent-injector-fc98d66b5   1         1         1       3h43m

NAME                     READY   AGE
statefulset.apps/vault   3/3     3h43m

$ k logs -n vault vault-0
==> Vault server configuration:

             Api Address: https://10.42.0.11:8200
                     Cgo: disabled
         Cluster Address: https://vault-0.vault-internal:8201
              Go Version: go1.19.2
              Listener 1: tcp (addr: "[::]:8200", cluster address: "[::]:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.12.1, built 2022-10-27T12:32:05Z
             Version Sha: e34f8a14fb7a88af4640b09f3ddbb5646b946d9c

2022-12-20T06:36:37.092Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2022-12-20T06:36:37.376Z [INFO]  core: Initializing version history cache for core
==> Vault server started! Log data will stream in below:

The problem is I can’t get access to the vault UI with Ingress!

The main ingress that I have configured in the values, returns 404.

I even write a custom ingress file for testing purposes:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.org/ssl-services: "vault-active"
  name: vault-ingress
  namespace: vault
spec:
  ingressClassName: nginx
  tls:
  - secretName: console.MYDOMAIN.COM
    hosts:
    - console.MYDOMAIN.COM
  rules:
    - host: console.MYDOMAIN.COM
      http:
        paths:
          - pathType: ImplementationSpecific
            backend:
              service:
                name: vault-ui
                port:
                  number: 8200
            path: /

$ k -n vault describe ing vault-ingress
Name:             vault-ingress
Labels:           <none>
Namespace:        vault
Address:          10.99.86.72
Ingress Class:    nginx
Default backend:  <default>
TLS:
  console.MYDOMAIN.COM terminates console.MYDOMAIN.COM
Rules:
  Host                   Path  Backends
  ----                   ----  --------
  console.MYDOMAIN.COM
                         /   vault-ui:8200 (10.36.0.6:8200,10.42.0.11:8200,10.44.0.7:8200)
Annotations:             nginx.org/ssl-services: vault-active
Events:
  Type    Reason  Age                From                      Message
  ----    ------  ----               ----                      -------
  Normal  Sync    34s (x2 over 38s)  nginx-ingress-controller  Scheduled for sync

But I can’t access the UI!

It gave me 404

Why is that?

K8s version: 1.25.3
helm version: v3.10.1

2

I suggest the problem may be that you have set ui.enabled=true in your Helm values, and are trying to use the vault-ui service.

This will seem counter-intuitive, but there does not appear to be any documentation for the ui.enabled feature in the Vault Helm chart, and as far as I can see, it is a remnant that was incorrectly left there when the Vault Helm chart was created by copying from the Consul Helm chart.

There is no reason I can see for there to be a separate ingress or service for the Vault UI.

As far as Vault is concerned, there is just one listener on port 8200 which serves both the API and UI together.

I propose that you delete the ui: section from your Helm values, delete your custom ingress, and simply access the UI directly via the main ingress you have configured in your Helm values.

3

Thank you @maxb for responding
The problem is I can’t even get the main Ingress that I have configured in my Helm values.
Both of them (MYDOMAIN.COM and console.MYDOMAIN.COM) are returning 404.

4

Hmm… actually, doesn’t that mean you’ve told your ingress to ingress no paths?

My config just says

server:
  ingress:
    enabled: true
    hosts:
      - host: MYDOMAIN.COM
6

I don’t think so:

helm install --dry-run --debug vault -n vault -f vault-values.yaml hashicorp/vault

[...]
# Source: vault/templates/server-ingress.yaml
apiVersion: networking.k8s.io/v1

kind: Ingress
metadata:
  name: vault
  namespace: vault
  labels:
    helm.sh/chart: vault-0.23.0
    app.kubernetes.io/name: vault
    app.kubernetes.io/instance: vault
    app.kubernetes.io/managed-by: Helm
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - "MYDOMAIN.COM"
      secretName: MYDOMAIN.COM
  ingressClassName: nginx
  rules:
    - host: "MYDOMAIN.COM"
      http:
        paths:

          - path: /
            
            pathType: Prefix
            
            backend:
              
              service:
                name: vault-active
                port:
                  number: 8200
[...]
7

Since your Vault is configured to speak TLS on port 8200, shouldn’t you have configuration in your ingress to tell it the backend speaks TLS?

I’m not sure how that being missing, would translate to a 404, though.

8

Yes, you are right.
But I have no idea how to configure it
I have searched a lot but I didn’t find anything…

9

Well, I’ve never done it either, but it didn’t take too much Googling to come across

nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"

Since TLS is the sticking point here, and you’re just doing some minikube testing, you might be best placed to just disable it (at every level of the stack) and check it works without it.

1 Like
10

Jesus F* christ!
Thank you so much, man!
You just saved me from a HUGE headache!
After adding this annotation to the vault values, It’s working like a champ!
Cheers🍻
I also mentioned you in the GitHub Issues and closed my issue.#18491.