Consul Replicate Permission Error

Consul
1

Hi,

We have a functional Consul cluster which contains KV stores and ACL / Tokens. I configured a new remote cluster for some testing. The two clusters see each other as different DC members through the wan gossip.

I wanted to replicate our KV stores from the current cluster to the new one. I am using Consul-Replicate to do this. The tool works, but I cannot replicate my KV Store. I keep getting 403 and ACL errors. I tried different solutions (using the same existing token on the first cluster, using the login / password, etc.) but that did not solve my problem.

Our initial need was only to replicate the KVs. And we don’t replicate ACLs from the first cluster to the second. Is it possible to make replication work with Consul-Replicate in this way? Or is it mandatory to replicate the ACL / Token from the first cluster to the second to have the KV stores replicated?

If not, I’m probably missing something on the way… Thank you for your help!

The errors I am getting look like:

[ERR] consul: RPC failed to server x.x.x.x:8300 in DC "fr": rpc error making call: ACL not found

[ERR] http: Request GET /v1/kv/xxxx?dc=fr&recurse=&stale=&wait=60000ms, error: rpc error making call: ACL not
consul: RPC failed to server x.x.x.x:8300 in DC "fr": rpc error making call: ACL not found

http: Request GET /v1/kv/xxxx?dc=fr&recurse=&stale=&wait=60000ms, error: rpc error making call: ACL not found from=127.0.0.1:51246

[ERR] consul: RPC failed to server x.x.x.x:8300 in DC "fr": rpc error making call: ACL not found

# consul-replicate -prefix "xxxx@fr" -consul-token="xxxxxxxxxxxxxxxx"
2021/12/16 12:14:32.979282 [WARN] (view) kv.list(xxxx@fr): Unexpected response code: 403 (retry attempt 1 after "250ms")
2021/12/16 12:14:33.243957 [WARN] (view) kv.list(xxxx@fr): Unexpected response code: 403 (retry attempt 2 after "500ms")

Thanks!

2

Consul replicate only supports configuring a single token for authenticating to Consul. It will use the same token to authenticate to both Consul datacenters.

If you do not want to enable token replication, you can alternatively create a token in the destination datacenter with the same accessor and secret IDs as the token you’re using to authenticate with the source DC.

$ consul acl token create -accessor="<accessor ID>" -secret="<secret ID>" \
  -policy-name="<policy with KV write permissions>"

This should resolve the ACL not found error you’re receiving, without requiring you to enable full token replication.

2 Likes
3

Hi Blake,

Thanks for your help and your response!

I had tried assigning the same token on both clusters but it didn’t work. I’m guessing I probably missed or misconfigured something.

Yesterday I configured ACL / Token replication and indeed it now works for KV replication through Consul-Replicate so maybe we will stay in that setup.

Anyway, I’m keeping your solution aside in case we remove the ACL replication part.

Maybe a sentence should be added about both of these solutions in the “Read Me” section of the Consul Replicate Github repository? It might help some people.

Thanks