Token replication is required for auth methods to function

Nomad
1

Hi,

I try to get Vault and Consul Workload Identity working, but .. its really confusing and much more complicated .. anyway

We have a two Consul Clusters, which replicates ACL and Tokens .. and I had to allow Consul Server access from for both DCs and Nomad Servers / Agents. So I was able to create the required stuff (role / acl / auth-methods ..) and I try now to get some example jobs running, but it fails, because:

Setup Failure: failed to setup alloc: pre-run hook "consul" failed: 1 error occurred: * failed to derive Consul token for service httpd: Unexpected response code: 500 (rpc error making call: Token replication is required for auth methods to function)

Also, we have two Nomad Clusters, which are not connected to each other.

The Nomad servers and Agents are configured, to use the local Consul Agent, which runs on the same node, like Nomad.

In other words:

On both Datacenters, we have a Nomad / Consul / Vault cluster, just Consul is connected each other, for replicate ACLs and Tokens, but nothing more.

Nomad Server Consul part looks like this:

...
 "consul": {
    "address": "dc-01-nomad-agent-ins-01.example.com:8501",
    "server_service_name": "nomad-ins",
    "client_service_name": "nomad-client-ins",
    "auto_advertise": true,
    "server_auto_join": true,
    "client_auto_join": false,
    "ssl": true,
    "ca_file": "/etc/ssl/private/chain.crt",
    "cert_file": "/etc/ssl/private/chain.crt",
    "key_file": "/etc/ssl/private/key",
    "token": "redacted"
  },
...

The primary Consul runs on the other DC, which has a different name …

dc-02-consul.example.com:8501 .. which means, there is no single URL for Consul, which points always to the primary cluster Consul Node …

The example howto I use:

So, I’m a bit lost ..

any suggestions?

cu denny