Vault not able to communicate to Consul over TLS

Vault
1

Hi,

I am trying to deploy Vault HA cluster on AWS EKS using Helm. There is already a Consul cluster that is deployed on same EKS cluster using Helm which is supposed to act as backend storage for Vault.

My Helm values file for Vault is below:

    server:

      extraVolumes:
      - type: secret
        name: consul-consul-ca-cert
      - type: secret
        name: consul-consul-ca-key
      - type: secret
        name: consul-consul-server-cert

      ha:
        enabled: true
        replicas: 3
        config: |
          ui = true

          listener "tcp" {
            tls_disable = 1
            address = "[::]:8200"
            cluster_address = "[::]:8201"
          }
          storage "consul" {
            address = "HOST_IP:8501"
            path = "vault"
            scheme = "https"
            tls_min_version = "tls12"
            tls_skip_verify = "false"
            tls_ca_file = "/vault/userconfig/consul-consul-ca-cert/tls.crt"
            tls_cert_file = "/vault/userconfig/consul-consul-server-cert/tls.crt"
            tls_key_file = "/vault/userconfig/consul-consul-server-cert/tls.key"
          }

          seal "awskms" {
            region     = "eu-west-1"
            kms_key_id = "AWS_KEY_HERE"
          }
        
      service:
        type: NodePort

      # Ingress allows ingress services to be created to allow external access
      # from Kubernetes to access Vault pods.
      ingress:
        enabled: true
        labels:
          name: vault-public
        annotations: | 
          external-dns.alpha.kubernetes.io/hostname: sandbox-vault.test.com
          alb.ingress.kubernetes.io/backend-protocol: HTTP
          alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
          alb.ingress.kubernetes.io/scheme: internet-facing
          alb.ingress.kubernetes.io/tags: ServiceOwner=Systems,Environment=sandbox,Name=vault-public
          alb.ingress.kubernetes.io/target-type: instance
          external-dns.alpha.kubernetes.io/ttl: "30"
          kubernetes.io/ingress.class: alb
        hosts:
          - host: sandbox-vault.test.com
            paths:
              - "/*"

    ui:
      # True if you want to create a Service entry for the Vault UI.
      #
      # serviceType can be used to control the type of service created. For
      # example, setting this to "LoadBalancer" will create an external load
      # balancer (for supported K8S installations) to access the UI.
      enabled: true
      serviceType: "NodePort"
      externalPort: 80

When the vault pods come up they give following error in the logs:
storage migration check error: error="Get https://10.110.1.45:8501/v1/kv/vault/core/migration: x509: certificate signed by unknown authority"

The certificates used by Consul are self-signed.

I have tried a lot of options but nothing seems to be working. Can someone please point me in the right direction?

Thanks

2

Hi,

I was able to fix this issue. Thanks for the help.

Thanks

3

I am having the same issue. How were you able to solve your issue?

4

@byronmansfield I’m a colleague of @Farhan1989. From what I recall of the issue, we were using the Consul server CA cert to establish trust. But $HOST_IP actually refers to the Consul client - an important distinction.

Solution is to add the Consul client CA cert to the container CA store. This can be done using consul-k8s per my instructions here:

Search in my post for the line “We also want to talk to the Consul client” - this is where I begin explaining retrieval of the client CA cert.

I hope this helps and saves you some time :slight_smile:

5

I have the same problem and checked your explanation.
I congratulate you but it looks like kind of your personal adventure.
I would expect HashiCorp develope a reasonable solution for that or stop pretending Vault can connect tls enabled Consul securely. I spent my days for that problem.