Can Consul be deployed with ACLs configured out of the box

Consul
1

I am builing a Consul cluster with Terraform and would like to pre-configure it as much as possible. Can ACLs be setup with the initial configuration or does it require a semi-manual bootstrap to be run after the cluster is up and running?

2

Hey @tvon,

Unfortunately, currently, there isn’t a way to do it through Consul. I could imagine doing it with Terraform though by using a local provisioner to call the bootstrap API and then save the bootstrap token somewhere secure. The tricky part is to make sure it gets called once Consul’s leader is elected. One way to accomplish that could be to make the local script continuously call the API with some timeout.

As a subsequent step, you could then use terraform to bootstrap the rest of the roles and tokens you’d need using the Consul terraform provider. I haven’t tried it though, so it’s just an idea :slight_smile:

1 Like
3

Thanks for the reply @ishustava

After some searching I figured that was the case and came up with something along the lines of what you are suggesting. I was initially just running the bootstrap once the server appeared to be ready but I eventually figured out that the best way to do this (and the best way to be safe and make sure it is run once on one node) was to poll the API until a server detected it had been elected leader.

It’s a first run at it, but seems like a reasonable approach.

For posterity, here be dragons:

Every server has this service (in addition to consul.service run as the recommended notify service):

# terraform template
# This may be over-engineered
[Unit]
Description="Consul acl bootstrap script"
Requires=consul.service
After=consul.service
# consul.service is type=notify so this will get triggered
PartOf=consul.service
StartLimitIntervalSec=30
StartLimitBurst=15

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/bin/consul-acl-bootstrap
ExecCondition=/usr/local/bin/consul-leader
Restart=always
Environment=ARM_SUBSCRIPTION_ID="${arm_subscription_id}"
Environment=ARM_TENANT_ID="${arm_tenant_id}"
Environment=ARM_CLIENT_ID="${arm_client_id}"
Environment=ARM_CLIENT_SECRET="${arm_client_secret}"

[Install]
WantedBy=consul.service

With consul being a notify service that should not run until consul is “ready”.

The consul-leader ensures the bootstrap is only attepted on a single node (kind of an unlikely edge case I guess but seems safer):

#!/usr/bin/env bash
#
# Test if this host is the cluster leader

ip=$(/bin/hostname -i)

leader=$(curl localhost:8500/v1/status/leader | sed -e 's/"//g' | cut -d: -f1)

test "$ip" == "$leader"

And consul-acl-bootstrap is (as a tf template) runs the bootstrap and a pre-defined policy:

#!/usr/bin/env bash
#
# terraform template
#

tempfile=$(mktemp)
acl_token=/etc/consul.d/acl-token
agent_token=/etc/consul.d/agent-token

vault_url=${vault_url}
vault_secret_name="${vault_secret_name}"

if consul acl bootstrap >"$tempfile"; then
  echo "bootstrapping..."
  CONSUL_HTTP_TOKEN=$(grep '^SecretID:' "$tempfile" | sed -r 's/SecretID: +([a-z0-9-]+)/\1/')
  echo -n "$CONSUL_HTTP_TOKEN" >"$acl_token"
  chmod 600 "$acl_token"

  export CONSUL_HTTP_TOKEN=$CONSUL_HTTP_TOKEN

  # Apply policies
  echo "applying baseline policy"
  consul acl policy create \
    -name "agent-token" \
    -description "Agent Token Policy" \
    -rules @/etc/consul.d/policies/agent-policy.hcl >/dev/null 2>&1

  echo "generating agent token"
  CONSUL_AGENT_TOKEN=$(consul acl token create \
    -description "Agent Token" \
    -policy-name "agent-token" | grep '^SecretID:' | sed -r 's/SecretID: +([a-z0-9-]+)/\1/')

  echo -n "$CONSUL_AGENT_TOKEN" >"$agent_token"
  chmod 600 "$agent_token"

  echo "acl token written to   $agent_token"
  echo "agent token written to $agent_token"
  
  # TODO: Actually post to KeyVault, curl whatver whatever.
fi

I think it might be a reach to make it all run in one terraform apply so I’m going with an initial run with bootstrap = true (that is used as a condition in some config templates) and a subsequent run with bootstrap = false that will fetch the configured KeyVault secret to use as the agent token and apply a rolling update to the consul server scale sets.

1 Like
4

Hi @tvon,

It is possible to bootstrap the ACL system using a predefined Secret ID. This is done by specifying a master token in the agent’s configuration file prior to starting Consul for the first time.

Here’s a quick example of this process.

  1. Create a well-known Secret ID
$ uuidgen
81984D92-2F32-49A3-A714-FD24E54EC391
  1. Configure this value under acl.tokens.master in your agent’s config file (e.g., server.hcl)
# server.hcl
server = true

acl {
    enabled = true
    default_policy = "deny"
    tokens {
        master = "81984D92-2F32-49A3-A714-FD24E54EC391"
    }
}
  1. Then start Consul.
consul agent -config-file=server.hcl

As Iryna suggested, this Secret ID can then be used by Terraform to create additional policies and tokens.

4 Likes
5

Thanks for the info, @blake.

Now I’m a bit torn between what seem like two routes (if I understand correctly):

  1. Having a pre-defined master token which would let me spin up an entire cluster in one go but I’d have to securely manage the token up-front somehow.
  2. Using a two-stage deploy (bootstrap = true then bootstrap = false) where all the tokens are generated and stored away from the prying eyes of humans.

The former seems more convenient but a smidge less secure.

I’ll just pick one and move on. I appreciate the feedback here.

6

Apologies for the noise, I might need more coffee. Or maybe less.

My initial goal here was to have one terraform run to get a fully secured consul cluster and various agent clusters. So far as I can tell that is not possible because I either need a cluster up and running before I can use the terraform provider to talk to it, or I need a cluster up and running before I can apply an agent policy and generate a token for that policy.

Based on the guide, the steps to setting up ACLs are:

  1. Add hcl config block.
  2. Run acl bootstrap to get an initial token
  3. Create a policy
  4. Use the generated token to apply the policy and get an agent token
  5. Add the agent token to the agents

When I was asking about pre-configuring the bootstrap I meant all of this. What is not apparently possible is:

  • #3 Defining a policy in server configs
  • #4 Defining a token attached to that policy.

Unless I am mistaken to get a fully functioning (secured) cluster I need multiple deploys. No big deal since I have the code for that now, I just wanted to clarify.