Vault Ingress sends HTTP request to HTTPS server

Vault
1

I have installed vault-helm chart v0.20.1 in an AKS 1.21 cluster. Vault is unsealed, auto-unseal works, the raft is stored in dynamodb, no problems there, but I am unable to forward requests from the ingress without disabling tls, and I would prefer to have encryption in transit throughout the stream (not passthrough and not terminated). The goal is to use self-signed certificates for intra-cluster communication while terminating the public ingresses with public-trusted certificates. All of which are in place, but I can’t get the ingress to speak to the vault service via tls. Seems silly to secure all internal app traffic with envoy proxies and intentions only to let vault traffic float around inside the cluster bare-naked :smiling_face_with_tear:

I am using nginx-ingress stable but no matter what annotations I provide the server.ingress in the Vault chart, I still have the proxy_pass schema as http, which seems to be causing the issue.

Error

curl -ki https://vault.testing.my-domain.com/
HTTP/1.1 400 Bad Request
Server: nginx/1.23.0
Date: Sat, 06 Aug 2022 20:23:04 GMT
Transfer-Encoding: chunked
Connection: keep-alive

Client sent an HTTP request to an HTTPS server.

Vault Helm Chart Values. vault-service is the internal self-signed certificate from the Consul CA Intermediary and vault-testing is a publicly trusted certificate pair for the ingress.

global:
  enabled: true
  tlsDisable: false

server:
  enabled: true
  extraInitContainers:
    - name: update-ca-certificates
      image: alpine
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: false
        runAsUser: 0
      command: [/bin/sh, -c]
      args:
        - apk add --no-cache ca-certificates && update-ca-certificates && cp -rv /etc/ssl/certs/* /etc/ssl/pods/
      volumeMounts:
        - name: ca-certs
          mountPath: /etc/ssl/pods
        - name: consul-ca-bundle
          mountPath: /usr/local/share/ca-certificates
          readOnly: true
  volumes:
    - name: ca-certs
      emptyDir: {}
    - name: consul-ca-bundle
      configMap:
        name: consul-ca-bundle
    - name: vault-service
      secret:
        secretName: vault-service
  volumeMounts:
    - name: ca-certs
      mountPath: /etc/ssl/certs
    - name: vault-service
      mountPath: /vault/userconfig/vault-tls/ca.crt
      subPath: ca.crt
      readOnly: true
    - name: vault-service
      mountPath: /vault/userconfig/vault-tls/vault.crt
      subPath: tls.crt
      readOnly: true
    - name: vault-service
      mountPath: /vault/userconfig/vault-tls/vault.key
      subPath: tls.key
      readOnly: true
  ingress:
    enabled: true
    annotations:
      nginx.ingress.kubernetes.io/proxy-ssl-secret: "vault/vault-service"
      nginx.ingress.kubernetes.io/proxy-ssl-verify: "off"
      # Other combinations of annotations I've tried
      #nginx.ingress.kubernetes.io/proxy-ssl-name: "vault"
      #nginx.ingress.kubernetes.io/ssl-passthrough: "true"
      #nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      #kubernetes.io/ingress.allow-http: "false"
      #nginx.ingress.kubernetes.io/configuration-snippet: |-
      #  proxy_pass https://vault-vault-vault.testing.my-domain.com-vault-active-8200;
    ingressClassName: nginx
    tls:
      - hosts:
          - vault.testing.my-domain.com
        secretName: vault-testing
    hosts:
      - host: vault.testing.my-domain.com
        paths:
          - /

  ha:
    enabled: true
    replicas: 3
    config: |
      ui = true
      listener "tcp" {
        tls_disable = 0
        tls_cert_file = "/vault/userconfig/vault-tls/vault.crt"
        tls_key_file  = "/vault/userconfig/vault-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-tls/ca.crt"
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "dynamodb" {
        ha_enabled     = "true"
        region         = "us-east-1"
        table          = "vault-raft"
      }
      seal "awskms" {
        region = "us-east-2"
        kms_key_id = "12345-54321-12345-54321"
      }
      service_registration "kubernetes" {}
      disable_mlock = true

ui:
  enabled: true

Nginx Ingress Pod Proxy-Pass Config

k8 exec -it pod/test-nginx-ingress-123456789-abcde -n ingress -- cat /etc/nginx/conf.d/vault-vault.conf
...
    proxy_pass http://vault-vault-vault.testing.my-domain.com-vault-active-8200;

Any insight would be most graciously appreciated.

2

I was able to resolve this and am providing the solution for posterity :face_with_monocle:

Turns out I was using the annotations from the wrong version of the Nginx Ingress. The docs for the version of the ingress I’m using also provided the location-snippets annotation required to add the internal-self-signed-client cert and ca to the ingress pod’s Nginx vhost config.

Minimally-configured, my vault helm-chart has this for the ingress section.

  ingress:
    enabled: true
    annotations:
      cert-manager.io/issuer: "vault-testing-internal"
      nginx.org/ssl-services: "vault-active"
      nginx.org/location-snippets: |
        proxy_ssl_certificate         /etc/nginx/secrets/ingress-service/tls.crt;
        proxy_ssl_certificate_key     /etc/nginx/secrets/ingress-service/tls.key;
        proxy_ssl_trusted_certificate /etc/nginx/secrets/ingress-service/ca.crt;

For the ingress to be a client to vault and use SSLv3 to connect to the upstream, I added these to the Nginx-Ingress helm-chart values after issuing a certificate for the Ingress under the same self-signed CA as Vault got it’s internal-server-certs from (so they can mutually authenticate within the k8s network). I also had to enable the use of snippets in the Nginx-Ingress.

  ## The volumes of the Ingress Controller pods.
  volumes:
    - name: ingress-service
      secret:
        secretName: ingress-service
        optional: false

  ## The volumeMounts of the Ingress Controller pods.
  volumeMounts:
    - name: ingress-service
      mountPath: "/etc/nginx/secrets/ingress-service/ca.crt"
      subPath: ca.crt
      readOnly: true
    - name: ingress-service
      mountPath: "/etc/nginx/secrets/ingress-service/tls.crt"
      subPath: tls.crt
      readOnly: true
    - name: ingress-service
      mountPath: "/etc/nginx/secrets/ingress-service/tls.key"
      subPath: tls.key
      readOnly: true

## Enable custom NGINX configuration snippets in Ingress, VirtualServer, VirtualServerRoute and TransportServer resources.
  enableSnippets: true

It should probably be noted that ssl-passthrough is probably “safer”. Since, in the use-case above if the Nginx-Ingress were compromised the container would be able to decrypt traffic destined for Vault and decrypt communication that may potentially include a vault-token (even though the ingress does not have a need a vault-token itself). In the alternative, SSL-Passthrough would expose the internal certificates at the Ingress, meaning the certs would not be trusted outside k8s without distributing the private-CA to external clients (something I wanted to avoid).