Hi guys, we are currently using consul v1.21.4 and we are planning to upgrade to latest version after few go related vulnerability introduced into the same version.
Recently not sure what is the issue I have just updated the version after v1.21.4 and none of them are working for me. Wanted to know what could be missing or newly introduced in the release.
I am getting below error :
Agent: Cannot discover address: cluster-LAN address=“providerd=aws region-eu-west-1 tag_key=consul_id tag_value-1ldevr1 service=ecs ecs_cluster=name-here” error-“discover-aws: Failed to get ECS Tasks: List Tasks failed: operation error ECS: ListTasks, https response error StatusCode: 403, Request ID deserialization failed, failed to decode response body, invalid character ‘<‘ looking for beginning of value”
I have changed nothing but just consul version from 1.21.4 to 1.21.5
I have below details in config.json file
Node_name
Client_addr
Data_dir
Retry_join : provider region tag_key tag_value service and ecs_cluster
Bind_addr
Bootstrap_expect
Not sure any of the above parameters as te absolute now with latest release, but these are the parameters that we used to provide with their values. I have only given keys in config file but it has their respective values too.
Kindly provide suggestions. Thank you
The error show the join strings:
“providerd=aws region-eu-west-1 tag_key=consul_id tag_value-1ldevr1 service=ecs ecs_cluster=name-here”
looks like there are multiple typos:
providerd should be provider
region-eu-west-1 should be region=eu-west-1
tag_value-1ldevr1 should be tag_value=1ldevr1
try fixing the typo first, and see how it goes!
This is the error
2025-12-04T10:52:49.442Z [ERROR] agent: Cannot discover address: cluster=LAN address="provider=aws region=Region_here tag_key=consul_id tag_value=tag_here service=ecs ecs_cluster=Cluster_name" error="discover-aws: Failed to get ECS Tasks: ListTasks failed: operation error ECS: ListTasks, https response error StatusCode: 403, RequestID: , deserialization failed, failed to decode response body, invalid character '<' looking for beginning of value" | -- | -- |
There was typo while posting here, as I wanted to remove some content, I might have added something else.
We already have the Listcluster, ListTask and describeTask action in our ECS task policy.
Is there anything changed in ecs:listTask command to get the task?
personally, running a stateful Consul cluster on ECS isn’t a good idea! Tasks in ECS can be replaced, IPs change, and storage is non-persistent by default.
it’ll make your cluster instability like leader elections triggered by task restarts, changing IP cause cluster panic, data loss on scale-in/redeploy. so it’s very hard to run in prod and troubleshoot on-the-fly.
but anyway, it’s your reference.