Consul backend tls connection problem

Vault
,
1

Hello,
I’ve installed Consul on Kubernetes and secured it via TLS.
Then I installed Vault and I would like to connect it to Consul.

Here is my config:

listener "tcp" {
        tls_disable = 1
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }
      storage "consul" {
        path = "vault/"
        address = "consul-consul-server.consul.svc.cluster.local:8501"
        scheme = "https"
        redirect_addr = "http://vault.vault.svc.cluster.local:8200"
        VAULT_ADDR = "http://vault.vault.svc.cluster.local:8200"
        token = "5f7cd758-6300-41234-a943-3be6d889601e"
      }

When I attempt to instal Vault via Helm, pods cannot be started; the following error message is displayed:

Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

I suspect Vault cannot connect to Consul via https.

Here my consul services:

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                                                   AGE
consul-consul-dns      ClusterIP   10.100.82.187   <none>        53/TCP,53/UDP                                                             3h3m
consul-consul-server   ClusterIP   None            <none>        8501/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8300/TCP,8600/TCP,8600/UDP   3h3m
consul-consul-ui       ClusterIP   10.100.169.26   <none>        443/TCP                                                                   3h3m

How can I connnect Vault to TLS enabled Consul?

Regards

2

Hello,
These are all secrets in Consul namespace:

NAME                                                       TYPE                                  DATA   AGE
consul-consul-acl-replication-acl-token                    Opaque                                1      15h
consul-consul-bootstrap-acl-token                          Opaque                                1      15h
consul-consul-ca-cert                                      Opaque                                1      15h
consul-consul-ca-key                                       Opaque                                1      15h
consul-consul-client-acl-token                             Opaque                                1      15h
consul-consul-client-token-zrwmx                           kubernetes.io/service-account-token   3      15h
consul-consul-gossip-encryption-autogenerate-token-s4sj5   kubernetes.io/service-account-token   3      11h
consul-consul-gossip-encryption-key                        Opaque                                1      15h
consul-consul-server-acl-init-cleanup-token-7qtm4          kubernetes.io/service-account-token   3      15h
consul-consul-server-acl-init-token-hpttl                  kubernetes.io/service-account-token   3      15h
consul-consul-server-cert                                  kubernetes.io/tls                     2      15h
consul-consul-server-token-99nr6                           kubernetes.io/service-account-token   3      15h
consul-consul-tls-init-token-7t52c                         kubernetes.io/service-account-token   3      11h
consul-tls                                                 kubernetes.io/tls                     3      15h
default-token-s7mxx                                        kubernetes.io/service-account-token   3      15h
sh.helm.release.v1.consul.v1                               helm.sh/release.v1                    1      15h
sh.helm.release.v1.consul.v2                               helm.sh/release.v1                    1      15h
sh.helm.release.v1.consul.v3                               helm.sh/release.v1                    1      11h
ui-tls                                                     kubernetes.io/tls                     3      15h

I changed consul config of Vault a bit but still no luck.

storage “consul” {
path = “vault/”
address = “consul-consul-server.consul.svc.cluster.local:8501”
scheme = “https”
redirect_addr = “http://vault.vault.svc.cluster.local:8200”
VAULT_ADDR = “http://vault.vault.svc.cluster.local:8200”
token = “5f7cd758-6300-483e-blabla-3be6d889601e”
tls_ca_file = “/vault/userconfig/consul-ca-cert/tls.crt”
tls_cert_file = “/vault/userconfig/consul-consul-server-cert/tls.crt”
tls_key_file = “/vault/userconfig/consul-consul-consul-key/tls.key”
}

Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

kubectl create secret generic consul-client-ca --from-literal=ca="$(kubectl exec consul-consul-server-0 -n consul -- curl -sk https://localhost:8501/v1/connect/ca/roots | jq -r .Roots[0].RootCert)" -n vault

and

      storage "consul" {
        path = "vault/"
        address = "consul-consul-server.consul.svc.cluster.local:8501"
        scheme = "https"
        redirect_addr = "http://vault.vault.svc.cluster.local:8200"
        VAULT_ADDR = "http://vault.vault.svc.cluster.local:8200"
        token = "5f7cd758-6300-483e-blabla-3be6d889601e"
        tls_ca_file = "/vault/userconfig/consul-client-ca/ca"
        tls_cert_file = "/vault/userconfig/consul-consul-ca-cert/tls.crt"
        tls_key_file  = "/vault/userconfig/consul-consul-ca-key/tls.key"
      }

Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

Which certs should I use?

What am I missing?

thanks&Regards

3

Are these self-signed certificates? If so, you either need to import the CA cert into the pods (rebuild them) or you have to turn off the SSL cert verify… I think it’s VAULT_SKIP_VERIFY for env but you have to check the docs to see how to turn it off in the config or helm.

4

Hello @aram
They are self-signed certificates were created by Consul.
I set it as below, but nothing changed:

extraEnvironmentVars:
    VAULT_SKIP_VERIFY: true

Thanks & Regards

5

Since it’s your first time setting up, turn off disable SSL completely on Consul and make sure that Vault can connect without it first.

6

@aram actually this is my second installation. things worked without tls. I enabled TLS at Consul and now Vault cannot connect to Consul.

Also, I looked for VAULT_SKIP_VERIFY but couldn’t find anything.

7

how can I dig deeper in Vault logs? I change many things but nothing changes the result; pulling my hairs here:

Readiness probe failed: Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused

the problem may not related with Consul connection, maybe it is related with Vault itself. How can I check Vault connects to Consul successfully or not?

Need to more than that error message.

8

@aram
I think the solution is here.

it’s a really big adventure. I don’t know people reluctant to accomplish so complex venture.

if there is no any other reasonable solution/way for about 19 months; against to the documentation, we may say…

"Vault cannot connect to tls enabled Consul securely?

Reagrds