I am just getting started with Consul via HCP. I’m trying to get a consul client to connect to the Consul cluster running on HCP. My issue at the moment is how to handle the ACL tokens. (As additional context, I’m using Terraform’s nomad_cluster module with install-consul and install-nomad).
I am curious about best practises for configuring the Consul client automatically.
The Consul HCP cluster is provisioned via Terraform. I am storing the consul_client_config and consul_ca_file in AWS Secrets Manager.
When the Consul client starts up, it pulls this consul_ca_file and consul_client_config from AWS Secrets Manager. This part works well. The part I am struggling with is how to automate the configuration of the agent ACL token.
I’ve taken a look at auto_config but I am struggling with seeing how to set that up with Consul HCP. I’ve also considered just creating the ACL token manually, then putting this in another AWS secret, which the client can also pull on boot.
Hi @jbye just some initial thoughts which come to mind after reading your scenario.
If you are using Terraform to create your HCP Consul cluster and generate the initial ACL root token, would it help in your use case/workflow, to then take that initial token to use with Consul Terraform Provider to generate the necessary ACLs tokens needed for your Consul clients using this resource and then pass the output to the resource which creates the client?
Utilizing the Consul Secrets Engine as part of the workflow could help too, the clients could request the token at startup time from Vault.
Hello,
I found your thread while trying to figure out the same.
it pulls this consul_ca_file and consul_client_config from AWS Secrets Manager. This part works well.
You confirm what I have seen in TF code written by Rosemary Wang in “hcp-examples”.
Here, TF pulls hcp_consul_client_token = hcp_consul_cluster.consul.consul_root_token_secret_id
out the root Token from the Consul cluster configuration in JSON retrieved from HCP hcp_consul_config_file = jsondecode(base64decode(hcp_consul_cluster.consul.consul_config_file))
and also saves it to AWS Secrets Manager along with the other secrets.
In my current understanding, this token is valid for up to 6h after the Consul cluster was created on HCP. So, if the Consul client is initialised within these first 6h hours, then it should be able to register with the server cluster.
Still, I will have to figure out if past these 6h, whether some manual action is required to retrieve a fresh token from HCP, or TF can also do this automagically using the hcp_consul_cluster_root_token resource for example?
After reading further into the TF code from Rosemary, it dawned on me that the root ACL token from HCP Consul cluster does not expire. At least not until a fresh one is generated by using hcp_consul_cluster_root_token. Also confirmed by running $ consul members -token "<root ACL token>" on one of my EC2 Consul & Vault client instances using the token I had retrieved from HCP some 3+ days ago after I had created a “dev” Consul cluster. Likely, I got confused by tokens for HCP Vault clusters.
Just to avoid that you also pull out a hair or two, note that Consul calls “SecretID” what Vault calls a “Token”.
From Understand the differences in token representations: "The way Vault and Consul refer to tokens in the command output is slightly different.", see the Table, and
“Note: The SecretID is used to authorize operations against Consul and should be generated from an appropriate cryptographic source.”.
So the code above does not pass the root ACL token “by reference” using its ID (as the ending of read-only .consul_root_token_secret_id appeared to stipulate (to me). But “by value” indeed, as SecretID is the root ACL token itself (which is also the reason why SecretID is marked as sensitive)!
