Consul Clients in ECS Fargate discovering servers but not connecting to them

Hi all,

I am running Consul servers on ECS EC2 which all connect up fine via retry-join on an NLB.

For the clients, I am using ECS Fargate and retry-join with the aws tags.

The clients seem to find the server instances and their IPv4 address and attempt to join them, but there’s no error logged about that failing. What happens is is that the client starts logging logs like:

2022-06-20T09:00:54.961Z [WARN] agent.router.manager: No servers available
2022-06-20T09:00:54.961Z [ERROR] agent: failed to sync changes: error="No known Consul servers".

I’ve seen a couple issues about the logs above in GitHub but no solutions and I can’t determine if this is even related.

Has anyone seen this before/know off the top of their head what this could be?

I’m actually looking into that now, did you manage to fix that?