Federation State RPC errors

Consul
1

We are currently seeing these federation RCP errors from one of our cluster leaders trying to reach another cluster:

[ERROR] agent.server.rpc: RPC failed to server in DC: server=xxx.xxx.xxx.150:8300 datacenter=us-west-2-prod method=FederationState.Apply error="rpc error making call: rpc error making call: Permission denied"
[WARN]  agent.replication.federation_state: replication error (will retry if still leader): error="failed to retrieve federation states: rpc error making call: Permission denied"

The cluster is otherwise healthy. ACLs are enabled in both this and the remote cluster (all running 1.8.4 community) and we haven’t had any problems with this cluster. To my knowledge, we have not configured anything related to WAN federation. Consul connect is also disabled.

As far as the ACL permissions go, this is what the agent is using. I can’t find anything in the docs that would tell me what ACL configuration is needed to make these calls succeed.

  operator = "read"
  agent_prefix "" {
    policy = "read"
  }
  event_prefix "" {
    policy = "read"
  }
  key_prefix "" {
    policy = "read"
  }		
  key_prefix "_rexec" {
    policy = "write"
  }
  key_prefix ".locks" {
    policy = "write"
  }
  node_prefix "" {
    policy = "write"
  }
  service_prefix "" {
    policy = "write"
  }
  session_prefix "" {
    policy = "write"
  }

We only just noticed these errors and don’t have any indication when they started due to log age. Any suggestions on what these errors mean and how to resolve them?

2

I was able to fix this issue myself. I ended up missing a step during ACL bootstrap. I had to run consul acl set-agent-token replication <token> with the ACL token in the secondary datacenter in order to clear these errors.

See ACL Replication for Multiple Datacenters | Consul - HashiCorp Learn