Helm Installed Vault with Consul Backend with Auto Encrypt and TLS Enabled using Consul Client

Consul
1

When Consul has TLS enabled, Auto Encrypt, and Consul Client enabled so that all K8s nodes have client via DaemonSet, Helm Installed Vault can’t use Consul Client as proxy to Consul as a Vault backend. Even with CA included via tls_ca_file, always get the following message,

[WARN] storage migration check error: error="Get "https://10.145.0.7:8501/v1/kv/vault/core/migration": x509: certificate signed by unknown authority"

Sample of config (from Vault Helm values file):

storage “consul” {
path = “vault/”
address = “HOST_IP:8501”
token = “XYZ”
scheme = “https”
tls_ca_file = “/vault/consul/tls/consul-ca-certs/ca.pem”
}

I have remoted into the vault pods and they can see the cert file. So, the CA cert file is there and I can’t see any other way to match the recommended configs, but I also can’t seem to riddle why this won’t work. The Consul Client for that node has no meaningful errors or issues in its logs. Everything looks good except that when Vault tries to access Consul via Consul Client (downward API, etc.) it can’t match up the certs.

For the record, if I add tls_skip_verify to the storage config, it works. So, there’s something going on where the client of the Consul Client (Vault in this case) doesn’t like the Consul Client cert. Seems like I must be doing something wrong though because it can’t be this hard… what does Vault need to have that it doesn’t have?

1 Like
2

Hi, sorry to hear this isn’t working. Can you share your values.yaml file for Consul and Vault?

3

Also can you verify that the ca.pem is the same as the contents of consul-ca-cert secret?

4

I have the same problem. Is there any solution for it?

I have Consul Servers running on EC2 instances. TLS/AutoEncrypt and Connect is enabled.

The client agents are running on EKS. They come up and get their certificates. But the Connect pods have problems to run.

It looks like that:
consul-54rzm 1/1 Running 0 112s
consul-connect-injector-webhook-deployment-86b759dfc5-6wczj 0/1 Init:0/1 0 112s
consul-connect-injector-webhook-deployment-86b759dfc5-rpnsh 0/1 Init:0/1 0 112s
consul-controller-6848c465c8-6w8c8 0/1 Init:0/1 0 112s
consul-g2pw2 1/1 Running 0 112s
consul-gjwcv 1/1 Running 0 112s
consul-sync-catalog-58d7f94fc4-sbwcq 0/1 Init:0/1 0 112s
consul-webhook-cert-manager-66bc8fb64f-9hvf6 1/1 Running 0 112s

The error message inside the get-auto-encrypt-client-ca container is the following:

Error retrieving CA roots from Consul: err=“Get “https://172.16.12.221:8501/v1/agent/connect/ca/roots”: x509: certificate signed by unknown authority”

5

Is it possible to get an update on this please?
I’m seeing this same issue in the latest version of the Helm chart, running Vault 1.10.0.

6

I am experiencing this issue and can confirm that the two are the same.
I can also confirm that the following is present in Vault’s storage stanza:
tls_ca_file = "/vault/tls/ca/tls.crt"
Given the error present and the fact that I have confirmed that the file at that path and the CA file in use by Consul match, it would appear that Vault 1.10.0 is ignoring the ca file parameter.