Using HCP Consul i have two DataCenters (Azure East and Azure West), traffice internal to the DC’s flows fine and I have the demo robot shop with about 8 services running fine. I am running Consul on Kubernetes connected to HCP as given in your GitHub learn-consul-get-started-kubernetes. I have two kubernetes clusters connected to the two HCP consul datacenters.
I have created a mesh gateway between the two DCs and I THINK it’s working.
For test I have killed the web service in East and am trying to get service mesh to use the one in west.
On the US east cluster I see:
However I am unable to get any traffic to flow:
I have tried:
While it shows up on the Consul UI as a service running remotely it doesn’t appear in the service API catalog.
I have also tried explicit annotations on the pod:
annotations: # CONSUL 'consul.hashicorp.com/connect-inject': 'true' #'consul.hashicorp.com/envoy-extra-args': '--log-level debug' # doesn't work this is already set 'consul.hashicorp.com/connect-service-upstreams': 'echo-1.svc.k8s-consul-federation-sbx-default.peer:3000'
Trying to get to that service results in error:
root@my-debug-container-74b66b994c-fs99t:/# curl http://127.0.0.1:3000
upstream connect error or disconnect/reset before headers. reset reason: connection failureroot@my-debug-container-74b66b994c-fs99t:/#
The gateway services is running: but doesn’t output any logs (like not a single line today):
➜ ~ kubectl --context aks-k8s-consul-fed-sbx-eus2-app1 get pods -n consul
NAME READY STATUS RESTARTS AGE
api-gateway-74c65b4f88-srdxf 1/1 Running 0 2d20h
consul-connect-injector-65bc5d97d-ktcqm 1/1 Running 0 2d21h
consul-mesh-gateway-579ffb95b7-656ns 1/1 Running 0 23h
consul-webhook-cert-manager-cb8546597-7ff7v 1/1 Running 0 3d
I can query the ‘remote’ service using the health API:
{{CONSUL_HTTP_ADDR}}/v1/health/service/web?peer=k8s-consul-federation-sbx-default
(works and finds the service)
This doesn’t work , no service found and why I think the “consul connect proxy” doesn’t work
{{CONSUL_HTTP_ADDR}}/v1/catalog/service/web.
I’m pretty fustrated, as I have a fair amount of consul experience, but I’ve spent the better part of a week on this and haven’t got a single packet across.
What debug should I do? How do I verify the mesh gateways are configured correct (I just enabled them in the Helm charts)