How to use prepared query in retry_join?

Consul
1

Outside using the way a prepared query can be accessed over DNS, is there a more direct way of putting in an expression in retry_join such that it uses prepared query in the same style of the way “provider” expressions work in retry_join?

I want to try using a prepared query to self discover consul peers in retry_join (assuming the consul server resolving the prepared query knows about said peers)?

2

This isn’t possible. In general the retry_join configuration needs to resolve to one or more addresses of Consul nodes in the cluster. If you knew where to direct the prepared query request to lookup the addresses then you would already have the address of a consul node and could just use that name/IP as the retry_join parameter directly instead of sending a prepared query request to that node.

Once joined to the cluster, restarting the same node shouldn’t require using the retry_join configuration again as the node will attempt to join any/all previously joined nodes.

3

Thank you for the response @mkeeler .

I asked a bad question with missing details. Let me try again

The plan I have is to use a single “bootstrap” Consul server that will be alive while the actual Consul cluster comes up.

The IP/hostname of this single “bootstrap” Consul server is known while the actual Consul cluster comes up.

I am wondering how I can configure the retry_join of the actual Consul cluster so that:

  • each node in the actual Consul cluster can discover each other, without me either having to hardcode the IPs of the actual Consul cluster nodes (hardcoding the IP of the single “bootstrap” Consul server should be OK)
  • or assuming the actual Consul cluster comes up on a specific cloud provider and then having to maintain separate code for each cloud provider

Would it be as simple as putting this in all the servers that form the actual Consul cluster:

retry_join = [bootstrap_consul_server]

I want to clarify that the bootstrap_consul_server will be torn down after the actual Consul cluster is up and running, so I would have to then update this with something that has the details of the new set of IPs

4

Yes you can just add the single known server as the only retry_join argument. As other nodes join the cluster, Consul will start tracking their IPs and will periodically emit a snapshot of the cluster membership status and store it into the data directory. If you restart and the bootstrap server is gone, it will still attempt to rejoin all previous nodes stored in that membership snapshot.

1 Like
5

Ah! I was missing this detail and sans that I was wondering how resiliency would work as retry_join would still be pointing to a dead-by-then bootstrap_consul_server

Appreciate this Matt

6

Cloud auto-join does simplify this for any cloud that is currently supported too. One trick I like to use is multiple config-file options. This enables me to split out the cloud-specific configuration and tends to make provisioning scripts easier since I can bake most of it as a static file in an image, and then add the dynamic elements as one or more separate files.

Click here to see an HCL example.

consul.hcl

bootstrap_expect = 3
client_addr = "0.0.0.0"
data_dir = "/opt/consul/data"
datacenter = "dc1"
node_name = "consul-server-1"
server = true
ui = true

acl {
  enabled                  = true
  default_policy           = "deny"
  enable_token_persistence = true
}

connect {
  enabled = true
}

dns_config {
  enable_truncate = true
}


ports {
  "grpc" = 8502
}

telemetry { 
  prometheus_retention_time = "1m" 
}

join-local.hcl

retry_join = ["10.0.2.21", "10.0.2.22", "10.0.2.23"]

join-aws.hcl

retry_join = ["provider=aws region=us-west-1 tag_key=node-role tag_value=consul-server" ]

(These are example tag keys and values.)

With that configuration, for local agents, you could run:

consul agent -config-file=consul.hcl -config-file=join-local.hcl

and for AWS instances:

consul agent -config-file=consul.hcl -config-file=join-aws.hcl

If you selectively deploy the configuration file with the retry_join attribute, you could place them in the same folder and use the -config-dir flag, rather than multiple -config-file flags too.

Using multiple configuration files can sure reduce the amount of repeated configuration or post-provisioning manipulation you need to do when you have elements that vary by environment.

Hopefully this helps you or a future reader, because it’s one of my favorite tricks that I think folks overlook.

-Charlie

1 Like
7

This is fantastic.

Where else have you documented this?

8

I’m happy that it seems useful to you! The -config-file and -config-dir flags for Consul configuration are documented on the Configuration page. You can even mix and match a directories and individual files using both.

For Nomad, the -config flag can be pointed to individual files or whole directories and you can supply as many as you need. The Nomad documentation goes into specific details about Nomad configuration load order and merging.

Additional information about Cloud Auto-Join can be found in:

1 Like