I’m in the process of upgrading Consul within our EKS dev cluster, going to version 1.15.7 from 1.13.9. This worked on a previous dev cluster but in this cluster I’m seeing an issue with the terminating-gateway config. For ease I will refer to them as eu-dev (working) and us-dev (broken).
In us-dev the consul-server pods are throwing errors about not being able to find the terminating-gateway config and when I connect to the consul ui I see 0 linked services next to terminating-gateway on the services page. Other than that everything appears to be fine and I don’t see any other errors.
The config should be managed with a CRD via the connect-injector but to simplify things during troubleshooting I have removed the CRD and am attempting to add the config manually via a json file and the consul config write command.
The config in eu-dev is identical apart from the datacenter name so if I take the output from that using consul config read -kind terminating-gateway -name terminating-gateway, put it into a json file and try to write it to the us-dev cluster with consul config write example.json I would expect everything to work. However this is where I get some odd behaviour.
As stated above the only difference in the config between clusters is the datacenter name which appears in a metadata field. If I leave that as eu-dev or remove it completely and write the config to the us-dev cluster everything works. If I set the datacenter to us-dev it doesnt. As soon as I write the config it appears to vanish. On subsequent consul config read commands I get:
Error reading config entry terminating-gateway/terminating-gateway: Unexpected response code: 404 (Config entry not found for "terminating-gateway" / "terminating-gateway")
I’m a bit lost as to what is happening here… any ideas?
Just to add a bit more info. Via the consul debug command I see this in the logs when I write the config and then cant read it as it seems to have been removed immediately. I’ve put a space around the lines of most interest, the first is where you see the PUT for me adding the config and the second is the Delete I believe.
2024-01-05T14:34:37.026Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:37.258Z [TRACE] agent.server: rpc_server_call: method=ConfigEntry.Apply errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=us-dev locality=local
2024-01-05T14:34:37.258Z [DEBUG] agent.http: Request finished: method=PUT url=/v1/config from=127.0.0.1:35584 latency=9.371116ms
2024-01-05T14:34:37.258Z [TRACE] agent.server: rpc_server_call: method=Internal.ServiceDump errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=true target_datacenter=us-dev locality=local
2024-01-05T14:34:37.258Z [TRACE] agent.server: rpc_server_call: method=Internal.ServiceDump errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=true target_datacenter=us-dev locality=local
2024-01-05T14:34:37.259Z [DEBUG] agent.http: Request finished: method=GET url="/v1/internal/ui/services?dc=us-dev&index=677363" from=127.0.0.1:60746 latency=1m4.235726528s
2024-01-05T14:34:37.259Z [DEBUG] agent.http: Request finished: method=GET url="/v1/internal/ui/services?dc=us-dev&index=677363" from=127.0.0.1:44060 latency=57.860477586s
2024-01-05T14:34:37.997Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:38.215Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:38.693Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:38.855Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:38.977Z [TRACE] agent.server: rpc_server_call: method=Status.Leader errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=us-dev locality=local
2024-01-05T14:34:38.977Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader?dc=us-dev from=127.0.0.1:42310 latency="120.545µs"
2024-01-05T14:34:39.026Z [TRACE] agent.server: rpc_server_call: method=Status.RaftStats errored=false request_type=read rpc_type=net/rpc leader=true
2024-01-05T14:34:39.071Z [TRACE] agent.server: rpc_server_call: method=ConfigEntry.Get errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=us-dev locality=local
2024-01-05T14:34:39.079Z [TRACE] agent.server: rpc_server_call: method=ConfigEntry.Delete errored=false request_type=write rpc_type=net/rpc leader=true target_datacenter=us-dev locality=local
2024-01-05T14:34:39.097Z [TRACE] agent.server: rpc_server_call: method=ConfigEntry.Get errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=us-dev locality=local
2024-01-05T14:34:39.424Z [TRACE] agent.server: rpc_server_call: method=Internal.ServiceDump errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=true target_datacenter=us-dev locality=local
2024-01-05T14:34:39.425Z [DEBUG] agent.http: Request finished: method=GET url="/v1/internal/ui/services?dc=us-dev&index=677564" from=127.0.0.1:44060 latency="908.603µs"
2024-01-05T14:34:39.819Z [TRACE] agent.server: rpc_server_call: method=Status.Leader errored=false request_type=read rpc_type=net/rpc leader=true allow_stale=false blocking=false target_datacenter=us-dev locality=local