Non-server agent unable to join secondary datacenter with "CA root is nil" error

davidschmitt · November 1, 2019, 1:48pm

For some reason I don’t understand I am getting the following error message when a client Consul agent attempts to join a secondary datacenter:

agent: AutoEncrypt failed: rpcinsecure error making call: rpcinsecure error making call: internal error: CA root is nil

The equivalent Consul configuration works in the primary datacenter.

Is this a problem with my configuration, or a limitation/bug of Consul?

Configuration overview:

Consul community edition version 1.6.1
Two Consul datacenters, one Consul domain
Using cloud auto-join (AWS).
Secondary datacenter Consul servers join the cluster successfully with healthy status
Secondary datacenter Consul servers have retry_join configured for secondary datacenter and retry_join_wan configured for primary datacenter
Failing client agent has retry_join configured for secondary datacenter (and no retry_join_wan)
Failing client agent is in the same subnet as the secondary datacenter Consul servers, with the same security groups applied
Gossip encryption enabled
TLS enabled/enforced
CA cert installed and configured on all hosts
ACL enabled/enforced
ACL persistence/replication enabled

Here are the exact consul.hcl configuration files I am using on the various nodes:

Failing client agent in secondary datacenter: backend.shi2.hcl.txt (1.7 KB)
Successful client agent in primary datacenter: frontend.shi1.hcl.txt (1.7 KB)
Server in primary datacenter: server.shi1.hcl.txt (1.9 KB)
Server in secondary datacenter: server.shi2.hcl.txt (1.9 KB)

ChipV223 · February 6, 2020, 3:58pm

Hi!

The behavior you are describing sounds a like a bug that was fixed in Consul v1.6.2 that didn’t allow auto_encrpyt to enable HTTPS on client nodes. You can read about the fix as well as the v1.6.2 changelog here:

github.com

hashicorp/consul/blob/master/CHANGELOG.md#162-november-13-2019

## UNRELEASED

IMPROVEMENTS:

* connect: Envoy's whole stats endpoint can now be exposed to allow integrations like DataDog agent [[GH-7070](https://github.com/hashicorp/consul/pull/7070)]

## 1.7.0-beta4 (January 31, 2020)

SECURITY

* agent: mitigate potential DoS vector allowing unbounded server resource usage from unauthenticated connections [[GH-7159](https://github.com/hashicorp/consul/issues/7159)]
* acl: add ACL enforcement to the `v1/agent/health/service/*` endpoints [[GH-7160](https://github.com/hashicorp/consul/issues/7160)]

IMPROVEMENTS:

* logging: Switch over to using go-hclog and allow emitting either structured or unstructured logs. [[GH-1249](https://github.com/hashicorp/consul/issues/1249)][[GH-7130](https://github.com/hashicorp/consul/pull/7130)]

BUGFIXES:

* acl: **(Consul Enterprise only)** `intention:write` privileges are now granted by the `namespace-management` policy that is injected into each new namespace.

This file has been truncated. show original