Documentation incorrect for auto_encrypt?

brianhays · February 20, 2020, 4:33pm

The consul documentation here indicates using "verify_incoming": true when setting up auto-encryption: https://learn.hashicorp.com/consul/security-networking/certificates#configure-the-clients

In practice, this results in the following error when attempting to start consul on a given agent/client:

[ERROR] agent: Error starting agent: error="VerifyIncoming set, and no Cert/Key pair provided! AutoEncrypt only secures the connection between client and server and doesn't affect incoming connections on the client."

The error message and the conversation in the following PR “seem” to indicate that the "verify_incoming": true setting on an agent is incompatible with auto-encryption: https://github.com/hashicorp/consul/pull/6811

Is this indeed the case? If so, what are the security implications of enabling auto-encryption and leaving "verify_incoming" set to false. Would this effectively disable mutual TLS between nodes?

brianhays · February 25, 2020, 5:36pm

I probably should have titled this something other than “Documentation” related - Does anyone know if the "verify_incoming": true setting on an agent is incompatible with auto-encryption? It seems this would effectively disable mutual TLS between nodes?

i0rek · March 2, 2020, 8:37pm

Thanks for reporting! This has been indeed a mistake in our guides and it is fixed now.

Mutual TLS between nodes in not affected by disabling verify_incoming on Consul clients. Since the only incoming requests for clients are from its own HTTP/HTTPS API. Since there is no way to extract an auto_encrypt certificate, it cannot be provided by a 3rd party tool querying the Consul client HTTPS API.

tvon · March 4, 2020, 3:16pm

This is not clear to me, the docs currently say:

{
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "consul-agent-ca.pem",
  "auto_encrypt": {
    "tls": true
  },
}

That config errors for me:

[ERROR] agent: Error starting agent: error="VerifyIncoming set, and no Cert/Key pair provided!"

Which makes sense with the discussion here, but if I remove verify incoming I get a segfault (UUIDs and IPs anonymized):

-- Logs begin at Tue 2020-03-03 22:58:41 UTC, end at Wed 2020-03-04 03:16:52 UTC. --
Starting "HashiCorp Consul - A service mesh solution"...
if auto_encrypt.allow_tls is turned on, either verify_incoming or verify_incoming_rpc should be enabled. It is necessary to turn it off during a migration to TLS, but it should definitely be turn
==> Starting Consul agent...
Version: 'v1.7.1'
Node ID: '00000000-0000-0000-0000-000000000000'
Node name: 'consul-ui-vm'
Datacenter: 'hashistack' (Segment: '')
Server: false (Bootstrap: false)
Client Addr: [127.0.0.9] (HTTP: 8500, HTTPS: -1, gRPC: -1, DNS: 8600)
Cluster Addr: 127.0.0.9 (LAN: 8301, WAN: 8302)
Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: false, Auto-Encrypt-TLS: true
==> Log data will now stream in as it occurs:
[DEBUG] agent: Using random ID as node ID: id=00000000-0000-0000-0000-000000000000
[DEBUG] agent.tlsutil: Update: version=1
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-ui-vm 127.0.0.9
[INFO]  agent: Started DNS server: address=127.0.0.9:8600 network=tcp
[INFO]  agent: Started DNS server: address=127.0.0.9:8600 network=udp
[INFO]  agent: Started HTTP server: address=127.0.0.9:8500 network=tcp
[INFO]  agent: Retry join is supported for the following discovery methods: cluster=LAN discovery_methods="aliyun aws azure digitalocean gce k8s linode mdns os packet
[INFO]  agent: Joining cluster...: cluster=LAN
[DEBUG] agent: discover: Using provider "azure": cluster=LAN
[WARN]  agent.client.manager: No servers available
[ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
[INFO]  agent: started state syncer
==> Consul agent running!
[DEBUG] agent: discover-azure: using vm scale set method. resource_group: rg-hashistack, vm_scale_set: consul-ss: cluster=LAN
[INFO]  agent: Sending GET https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/microsoft.Compute/vi
[INFO]  agent: GET https://management.azure.com/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/microsoft.Compute/virtualMac
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Interface /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-hashistack/providers/Microsoft.Compute/virtualMachineSc
[DEBUG] agent: discover-azure: Found ip addresses: [127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9]: cluster=LAN
[INFO]  agent: Discovered servers: cluster=LAN cluster=LAN servers="127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9 127.0.0.9"
[INFO]  agent: (LAN) joining: lan_addresses=[127.0.0.9, 127.0.0.9, 127.0.0.9, 127.0.0.9, 127.0.0.9, 127.0.0.9, 127.0.0.9]
[DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.9:8301
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-ss-vm000001 127.0.0.9
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-ss-vm000005 127.0.0.9
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-ss-vm000003 127.0.0.9
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: consul-ss-vm000000 127.0.0.9
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: appservers-vm000002 127.0.0.9
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: appservers-vm000000 127.0.0.9
[DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.9:8301
[DEBUG] agent.client.memberlist.lan: memberlist: Failed to join 127.0.0.9: dial tcp 127.0.0.9:8301: connect: connection refused
[DEBUG] agent.client.memberlist.lan: memberlist: Initiating push/pull sync with: 127.0.0.9:8301
[INFO]  agent.client: adding server: server="consul-ss-vm000001 (Addr: tcp/127.0.0.9:8300) (DC: hashistack)"
[INFO]  agent.client: adding server: server="consul-ss-vm000005 (Addr: tcp/127.0.0.9:8300) (DC: hashistack)"
[INFO]  agent.client: adding server: server="consul-ss-vm000003 (Addr: tcp/127.0.0.9:8300) (DC: hashistack)"
[INFO]  agent.client: adding server: server="consul-ss-vm000000 (Addr: tcp/127.0.0.9:8300) (DC: hashistack)"
[DEBUG] agent.tlsutil: OutgoingRPCWrapper: version=1
[DEBUG] agent.tlsutil: OutgoingRPCConfig: version=1
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x4602ac]
goroutine 24 [running]:
crypto/tls.(*clientHandshakeStateTLS13).sendClientCertificate(0xc0004d8dc8, 0x0, 0x0)
/usr/local/go/src/crypto/tls/handshake_client_tls13.go:540 +0x129
crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc0004d8dc8, 0xc0005805a0, 0x0)
/usr/local/go/src/crypto/tls/handshake_client_tls13.go:91 +0x230
crypto/tls.(*Conn).clientHandshake(0xc0003dee00, 0x0, 0x0)
/usr/local/go/src/crypto/tls/handshake_client.go:198 +0x49e
crypto/tls.(*Conn).Handshake(0xc0003dee00, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1362 +0x12b
crypto/tls.(*Conn).Write(0xc0003dee00, 0xc0007aef68, 0x1, 0x1, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1094 +0xb7
github.com/hashicorp/consul/agent/pool.(*ConnPool).getNewConn(0xc000226150, 0xc000134d71, 0xa, 0x35ebb00, 0xc0005e1b60, 0x2, 0x7fb15d6f3401, 0x0, 0x11, 0xc000264568)
/home/circleci/project/consul/agent/pool/pool.go:349 +0x118
github.com/hashicorp/consul/agent/pool.(*ConnPool).acquire(0xc000226150, 0xc000134d71, 0xa, 0x35ebb00, 0xc0005e1b60, 0x2, 0xc000226101, 0x10, 0xc0007691b8, 0x47c352)
/home/circleci/project/consul/agent/pool/pool.go:228 +0x4d1
github.com/hashicorp/consul/agent/pool.(*ConnPool).getClient(0xc000226150, 0xc000134d71, 0xa, 0x35ebb00, 0xc0005e1b60, 0x2, 0x33fe8b01, 0x1, 0xc00004c3c0, 0x1, ...)
/home/circleci/project/consul/agent/pool/pool.go:406 +0x9b
github.com/hashicorp/consul/agent/pool.(*ConnPool).rpc(0xc000226150, 0xc000134d71, 0xa, 0x35ebb00, 0xc0005e1b60, 0x2, 0x3066688, 0x17, 0x1, 0x2fcc2e0, ...)
/home/circleci/project/consul/agent/pool/pool.go:462 +0xb6
github.com/hashicorp/consul/agent/pool.(*ConnPool).RPC(0xc000226150, 0xc000134d71, 0xa, 0x35ebb00, 0xc0005e1b60, 0x2, 0x3066688, 0x17, 0x1, 0x2fcc2e0, ...)
/home/circleci/project/consul/agent/pool/pool.go:432 +0xfa
github.com/hashicorp/consul/agent/consul.(*Client).RPC(0xc000336080, 0x3066688, 0x17, 0x2fcc2e0, 0xc0001ace80, 0x2d4fd80, 0xc000535d60, 0x0, 0x0)
/home/circleci/project/consul/agent/consul/client.go:314 +0x1f1
github.com/hashicorp/consul/agent/local.(*State).updateSyncState(0xc0007601c0, 0x0, 0x0)
/home/circleci/project/consul/agent/local/state.go:847 +0x222
github.com/hashicorp/consul/agent/local.(*State).SyncFull(0xc0007601c0, 0x0, 0x1b)
/home/circleci/project/consul/agent/local/state.go:1014 +0x2b
github.com/hashicorp/consul/agent/ae.(*StateSyncer).nextFSMState(0xc000582000, 0x3037e29, 0x8, 0x3037e29, 0x8)
/home/circleci/project/consul/agent/ae/ae.go:176 +0x46e
github.com/hashicorp/consul/agent/ae.(*StateSyncer).runFSM(0xc000582000, 0x3037e29, 0x8, 0xc0004d9fb8)
/home/circleci/project/consul/agent/ae/ae.go:162 +0x3a
github.com/hashicorp/consul/agent/ae.(*StateSyncer).Run(0xc000582000)
/home/circleci/project/consul/agent/ae/ae.go:156 +0x7c
created by github.com/hashicorp/consul/agent.(*Agent).StartSync
/home/circleci/project/consul/agent/agent.go:1951 +0x43

This is a new cluster that I’m regularly tearing down and rebuilding with Terraform. It is not clear to me what the initial configuration should look like.

If this should be a github issue or it’s own thread I will happily comply.

i0rek · March 6, 2020, 9:11pm

I apparently missed another mistake in our guides! I just made a PR so that it will be fixed online soon! Thanks for reporting!

The segfault should definitely not be happening! I created a github issue for you with the information you already provided: https://github.com/hashicorp/consul/issues/7407. I will check that soon.

i0rek · March 9, 2020, 10:38am

The following line:

if auto_encrypt.allow_tls is turned on, either verify_incoming or verify_incoming_rpc should be enabled. It is necessary to turn it off during a migration to TLS, but it should definitely be turn

doesn’t match your configuration. Is this segfault from a server? I think you provided a client configuration. Could you respond on the github issue if possible?

tvon · March 11, 2020, 3:54am

Apologies for not getting back to you, this is probably pebkac but I haven’t had a chance to verify. I will update the GitHub issue as soon as I can.