X509: certificate signed by unknown authority on Consul client

Consul
1

I just have TLS enabled for the HTTP API, but I can’t figure out this certificate error with auto_encrypt enabled when I run consul members on Consul clients, it gives me x509: certificate signed by unknown authority.

I follow this tutorial to generate certs on my workstation. I also run sudo update-ca-certificates to add the CA cert to /etc/ssl/certs/ on each node.

Part of the settings for the Consul client:

server = false
datacenter = "dc1"
data_dir = "/opt/consul"
client_addr = "127.0.0.1"

verify_incoming = false

verify_outgoing = true
verify_server_hostname = true
ca_file = "/usr/local/share/ca-certificates/custom/consul-agent-ca.crt"
#cert_file = "/opt/consul/cert/dc1-client-consul-0.pem"
#key_file = "/opt/consul/cert/dc1-client-consul-0-key.pem"
auto_encrypt = {
  tls = true
}
encrypt ="xxx"
connect {
  enabled = true
}
ports{
  https = 8501
  http  = -1
}

If I uncomment cert_file and key_file, the consul members command would work, but that doesn’t seem to be how auto_encrypt is supposed to work. Is there any settings I’m missing to solve this issue?

Settings for Consul Server:

server = true
datacenter = "dc1"
data_dir = "/opt/consul"
client_addr = "127.0.0.1"
verify_incoming = true
verify_outgoing = true
verify_server_hostname = true
ca_file = "/usr/local/share/ca-certificates/custom/consul-agent-ca.crt"
cert_file = "/opt/consul/cert/dc1-server-consul-0.pem"
key_file = "/opt/consul/cert/dc1-server-consul-0-key.pem"
auto_encrypt {
  allow_tls = true
}
encrypt="xxx"
connect {
  enabled = true
}
ports{
  https = 8501
  http  = -1
}

/etc/environment:

CONSUL_HTTP_SSL=true
CONSUL_HTTP_ADDR=127.0.0.1:8501
# These two are needed for Consul servers to use `consul members`
#CONSUL_CLIENT_KEY=/opt/consul/cert/dc1-client-consul-0-key.pem
#CONSUL_CLIENT_CERT=/opt/consul/cert/dc1-client-consul-0.pem
2

I just tried this suggestion from Consul Auto Encrypt: Client Certificate says "x509: certificate signed by unknown authority" · Issue #8636 · hashicorp/consul · GitHub

curl -k  https://127.0.0.1:8501/v1/connect/ca/roots | jq -r '.Roots[]."RootCert"' >> /ca.cert

export CONSUL_CACERT=/ca.cert

It works on Consul client. I can run consul members now without issues. But is this the only way to solve this? Is my CA cert generated by consul tls ca create no longer used when the auto_encrypt is enable?

3

Hi @laukaichung,

When using auto_encrypt, the certificates for clients are signed by Consul Connect CA and not the CA generated using Consul TLS utility.

Ref: Configuration | Consul by HashiCorp

You still have to have ca_file in client configurations so that the client can verify the remote Consul Server certificates.

1 Like
4

Thank you for the answer.
Here’s what I found. If I want to issue any consul commands on a client node with TLS enabled, I need to either

  1. download the root cert from https://127.0.0.1:8501/v1/connect/ca/roots and export its path to the environment.

Or

  1. In the config file, set the cert_file and key_file with the certs generated by Consul TLS utility

    server = false
    datacenter = “dc1”
    data_dir = “/opt/consul”
    client_addr = “127.0.0.1”
    verify_incoming = false
    verify_outgoing = true
    verify_server_hostname = true
    ca_file = “/usr/local/share/ca-certificates/custom/consul-agent-ca.crt”
    cert_file = “/opt/consul/cert/dc1-client-consul-0.pem”
    key_file = “/opt/consul/cert/dc1-client-consul-0-key.pem”
    auto_encrypt = {
    tls = true
    }

However, using CONSUL_CLIENT_KEY and CONSUL_CLIENT_CERT env never work for me. Do you know why?

5

The 2nd option you used here is manual certificate management. auto_encrypt feature is introduced to get rid of the overhead of managing manual certificates. When you have cert_file and key_file defined there is no need of auto_encrypt { tls = true }.

CONSUL_CLIENT_KEY and CONSUL_CLIENT_CERT env is used when you have mTLS authentication enabled by setting verify_incoming = true. In your case verify_incoming is set to false, so passing client certificate and key wont work.

1 Like
6

Is manually getting the Connect’s built in CA cert from the API and saving it on every client node the only way of fixing this issue for now?

1 Like
7

There is one more option that you could use but is usually not considered as a best practice from a security standpoint in some scenarios.

But considering that you are talking to a service running on localhost and if you are not worried too much about the CA who signed the certificate of the agent you can disable SSL certificate validation by setting the CONSUL_HTTP_SSL_VERIFY variable to false.

By doing the above you don’t have to extract the connect CA and set the CONSUL_CACERT variable to talk to Consul.

1 Like