How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)

Vault
, , ,
1

Requesting for some help on : How to connect Vault with Consul agent on Kubernetes via Helm chart (Consul server is on Azure managed app)

I’m trying to build a POC, for Vault and consul and got some questions.

  1. Deployed Azure managed app using - Deploy HashiCorp Consul Service on Azure | Consul - HashiCorp Learn

  2. Installed consul agent on AKS with the steps in the Connect an Azure Kubernetes Service (AKS) cluster to HCP on Azure | Consul - HashiCorp Learn
    Consul helm chart : GitHub - hashicorp/consul-helm: Helm chart to install Consul and other associated components.

  1. Installed vault via helm chart: GitHub - hashicorp/vault-helm: Helm chart to install Vault and other associated components.

Kubernetes services and pods for consul.

~$kubectl get svc -n consul                                            
NAME                          TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
consul-connect-injector-svc   ClusterIP   10.0.252.97   <none>        443/TCP   2d13h
consul-controller-webhook     ClusterIP   10.0.169.80   <none>        443/TCP   2d13h

~$kubectl get pods -n consul
NAME                                                         READY   STATUS    RESTARTS   AGE
consul-27j4j                                                 1/1     Running   0          2d13h
consul-connect-injector-webhook-deployment-9454b8d68-778rd   1/1     Running   0          2d13h
consul-controller-7857456f99-mhzpw                           1/1     Running   1          2d13h
consul-lkhpl                                                 1/1     Running   0          2d13h
consul-webhook-cert-manager-cfbb689f7-fgtlw                  1/1     Running   0          2d13h
consul-zf989                                                 1/1     Running   0          2d13h

vault config as below:

ui:
  enabled: true
  serviceType: LoadBalancer
server:
  ingress:
    enabled: true
    extraPaths:
      - path: /
        backend:
          serviceName: vault-ui
          servicePort: 8200
    hosts:
      - host: vault.something_masked.com
  ha:
    enabled: true
    config: |
      ui = true

      listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "consul" {
        path = "vault/"
        scheme = "https"
        address = "HOST_IP:8500"
      }

Vault pods

kubectl get pods -n vault                             
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                0/1     Running   0          7m14s
vault-1                                0/1     Running   0          7m11s
vault-2                                0/1     Running   0          7m11s
vault-agent-injector-cbbb6f4df-rmbd7   1/1     Running   0          7m22s

ERROR : Vault is unable to make communication with consul agent.

Logs for vault-0 pod

kubectl logs vault-0 -n vault

WARNING! Unable to read storage migration status.
2021-06-27T08:37:17.801Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2021-06-27T08:37:18.824Z [WARN]  storage migration check error: error="Get "https://10.54.0.206:8500/v1/kv/vault/core/migration": dial tcp 10.54.0.206:8500: connect: connection refused"

Logs for vault-agent-injector pod

 kubectl logs vault-agent-injector-cbbb6f4df-rmbd7 -n vault    
2021-06-27T08:37:09.189Z [INFO]  handler: Starting handler..
Listening on ":8080"...
2021-06-27T08:37:09.218Z [INFO]  handler.auto-tls: Generated CA
2021-06-27T08:37:09.219Z [INFO]  handler.certwatcher: Updated certificate bundle received. Updating certs...
2021-06-27T08:37:18.252Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s
2021-06-27T08:37:18.452Z [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=10s

Any suggestions or advice in the vault configuration if I have missed something ??

Thank you in advance.

Regards
Pooja

2

Hi @ishustava1 @ishustava
Could you please help me with the above issue.

thanks in advance,
Pooja

3

Hey @sanaka.pooja4

Sorry for the delayed response.

Consul service on Azure has ACLs and TLS with auto-encrypt enabled. As a result you need to first provide the right TLS configuration to vault. There are steps for it described here. The difference is that you need to get the CA from the HCS consul server URL instead of exec’ing into the consul-server pod.

In addition, you need to provide an ACL token for vault. There are brief instructions here.

Hope this helps!