Hello all
My question is that in case of federated environment if services are submitted as Nomad jobs, the nomad together with consul creates the Service Identity tokens to be able to communicate with them (also with sidecar proxies). These tokens are created as a local ones (makes sense, that we don’t need to replicate them). These local service identities tokens are created on the leader of the nomad servers. And here the problem starts. If the job is submitted to the dc1 and the leader of nomad servers is server from dc2, these tokens aren’t readable at all. The errors on the dc1 is “rpc error making call: token does not exist: ACL not found”. My question is: Is it possible somehow to tell nomad also to create these tokens on the dc, which is used for the job deployment?
@martin.seleng Not sure I understand your environment, but I can say that all Nomad servers of a given Nomad region need to be talking to Consul agents of the same Consul dc. Otherwise you’ll have unrecognized ACL problems, like you describe.
@seth.hoenig let me start once again. i have an 3 different datacenters in one region (WNA federation) and their consul servers are connected by gateways (Mesh Gateways in local mode). The consul itself is working as expected. On every datacenter there is a local communication through the local interface of the gateway and between datacenters the public interfaces are used. One of the dc is marked as primary and replication is enabled. So if the token is created as global, it is immediately replicated to other datacenters. Consul is working as expected. Now imagine this scenario: you had one service deployed on dc1 and one on dc2 and they want to communicate. No problem if you ran them as services directly by consul. You create tokens with right permissions (for services) and also runs the envoy sidecar proxies manually. Everything again works as expected. An now what I want to achieve is to submit them as a nomad jobs. One of them to dc1 and second to dc2. Nomad creates for me automatically si_tokens (server identities) and tries to start the services with envoy sidecar proxies, but here is the problem. The tokens are created as Local (there is big discussion why it should be local and why it was changed). Totally agree, their do not to be global, but the problem is, that if you submit the job to be deployed on dc1, it will land there and if dc2 it will so be deployed on dc2, but the si_tokens are created as local only on nomad server leader, not on the dc i am submitting the jobs. And this makes perfect sense of the error too, but doesn’t solve my problem. It can’t see the tokens at all as they are local and created on nomad server leader. Hope it helps