Enabling tls - certificate signed by unknown authority

Consul
1

Ok trying to enable tls on consul server. Can you please help point out what I am doing wrong? The config works just fine in http mode.

env values:

CONSUL_HTTP_ADDR=localhost:8501
CONSUL_HTTP_SSL=true

Consul cert dir contents:

-rw-r--r-- 1 consul consul  227 Nov  2 23:54 consul-agent-ca-key.pem
-rw-r--r-- 1 consul consul 1078 Nov  2 23:54 consul-agent-ca.pem
-rw-r--r-- 1 consul consul  227 Nov  2 23:54 miami-server-consul-0-key.pem
-rw-r--r-- 1 consul consul  973 Nov  2 23:54 miami-server-consul-0.pem

The ca was created with:
consul tls ca create

The cert was created with on the same machine:
consul tls cert create -server -dc miami

When I list the ports:

consul    194486          consul    6u  IPv6 300411      0t0  TCP *:8300 (LISTEN)
consul    194486          consul    8u  IPv6 300412      0t0  TCP *:8302 (LISTEN)
consul    194486          consul   12u  IPv6 300414      0t0  TCP *:8301 (LISTEN)
consul    194486          consul   14u  IPv4 282393      0t0  TCP 10.10.10.1:8501 (LISTEN)
consul    194486          consul   15u  IPv4 282394      0t0  TCP 100.x.x.x:8501 (LISTEN)
consul    194486          consul   17u  IPv4 306372      0t0  TCP 127.0.0.1:8600 (LISTEN)
consul    194486          consul   20u  IPv4 311307      0t0  TCP 10.10.10.1:8500 (LISTEN)
consul    194486          consul   21u  IPv4 311308      0t0  TCP 100.x.x.x8500 (LISTEN)
consul    194486          consul   22u  IPv4 311309      0t0  TCP 127.0.0.1:8500 (LISTEN)
consul    194486          consul   23u  IPv4 282395      0t0  TCP 127.0.0.1:8501 (LISTEN)

when I run the command on the machine I just created the certs:
consul info

I get the following in the cli:
Error querying agent: Get "https://localhost:8501/v1/agent/self": x509: certificate signed by unknown authority

I have read the documentation and it only talks about making sure you do it all on the same machine which I have done.

Here is my working config:

node_meta = {
    "instance_type" = "Dual Silver",
    "location" = "Miami"
  }

server = true

bootstrap_expect = 3

ui_config {
  enabled = true
}

datacenter = "miami"
data_dir   = "/etc/consul.d/data"
log_level  = "INFO"

# this will grab the ip dynamically from the interface name.
bind_addr = "0.0.0.0"

addresses {
  # Defaults to the first private IP address.
  http =  "{{ GetInterfaceIP \"enp94s0f1\" }} {{ GetInterfaceIP \"tailscale0\" }} 127.0.0.1"
  https  = "{{ GetInterfaceIP \"enp94s0f1\" }} {{ GetInterfaceIP \"tailscale0\" }} 127.0.0.1"
  grpc = "{{ GetInterfaceIP \"enp94s0f1\" }}"
}

advertise_addr = "{{ GetInterfaceIP \"enp94s0f1\" }}"

retry_join = ["host1.vpc","host2.vpc","host3.vpc"]

# service mesh switch
connect {
  enabled = false
}

# Security Stanza
encrypt                = "redacted"
verify_incoming        = false
verify_outgoing        = false
verify_server_hostname = false
ca_file                = "/etc/consul.d/cert/consul-agent-ca.pem"
cert_file              = "/etc/consul.d/cert/miami-server-consul-0.pem"
key_file               = "/etc/consul.d/cert/miami-server-consul-0-key.pem"
ports {
  https = 8501
}
4

I thought I had a solution, but I didn’t. When I read the error it is telling me that the cert was signed by an unknown authority. The ca pair was in the same directory when I issued the cert.

Please any help would be appreciated.
Brad

5

That error indicates that the command is reaching the server, but communication is being rejected because the TLS handshake can’t be negotiated. Try setting the environment variable CONSUL_CACERT to where the CA file is located Example: export CONSUL_CACERT=/etc/consul.d/cert/consul-agent-ca.pem

@bradley I’m not sure if you are aware of this resource Consul Tutorials - HashiCorp Learn
but it’s very helpful as you are navigating the Consul world.

6

Yes that was the issue. Doh! So here is what confused me. I am running 3 servers, and was able to have them all talk tls to each other. Then when I went to try the cli to see the health etc, it was failing. After banging my head for hours… I finally realized that the cli is a CLIENT. The client needs client certs to talk to the server. So I generated the certs, and viola it works. Here is what I put in my /etc/environment to make the cli work for any user on the host

CONSUL_HTTP_ADDR=localhost:8501
CONSUL_HTTP_SSL=true
CONSUL_CLIENT_CERT=/etc/consul.d/cert/miami-client-consul-2.pem
CONSUL_CLIENT_KEY=/etc/consul.d/cert/miami-client-consul-2-key.pem
CONSUL_CACERT=/etc/consul.d/cert/consul-agent-ca.pem

I have to admit the process was painful, the docs are good, however scattered and hard to find the necessary bits. I am going to do a video on how I did this and will post a reply here with the link so others can benefit from it.

Thanks for the help.
Brad

1 Like
7

I’m glad to hear you got it to work and yes the concept of the binary being an agent and server (depending on configuration) can be abit confusing. Two weeks ago we refreshed the agent overview page doc page. If possible, I’d love your feedback. In the meantime, keep sending us questions and we’ll do our best to answer them.

8

OK, not to be critical, but your documentation is inconsistent in one particular area. This is the creation and use of TLS for nomad, vault, and consul.

If you look at the docs, it suggests the built in ca cert generator within consul. Ok this worked great. However if you go to the nomad docs, it suggests cfssl and a different method. Then when you look at vault it doesn’t even talk about how to do it.

I understand that vault has the ability to issue certs. But that is after it is installed.

Also your configuration key names differ from product to product. Let me show you

Consul:
ca_file = “/etc/consul.d/cert/consul-agent-ca.pem”
cert_file = “/etc/consul.d/cert/miami-server-consul.pem”
key_file = “/etc/consul.d/cert/miami-server-consul-key.pem”

nomad:
ca_file = “/etc/nomad.d/cert/consul-agent-ca.pem”
cert_file = “/etc/nomad.d/cert/server.pem”
key_file = “/etc/nomad.d/cert/server-key.pem”

Vault:
tls_cert_file = “/etc/vault.d/cert/server.pem”
tls_key_file = “/etc/vault.d/cert/server-key.pem”
tls_client_ca_file = “/etc/vault.d/cert/consul-agent-ca.pem”

Why would you do that? You just make it harder for the new people to install the product.

The whole point of zero trust is to use tls right out of the box. There should be a video and text guide on how to set this up on all three installed as a stack.

This by far has been the hardest part of the install. I am not there yet still trying to get vault setup on tls.

Brad

9

Agree, this part is vague in the document.