Error on Creating Service Router

parag · June 26, 2021, 9:44pm

I created a Service Router (on a kubernetes cluster) using the following yaml.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: fmrouter
spec:
  routes:
    - match:
        http:
          pathPrefix: '/iam'
      destination:
        service: kc-wrap
    - match:
        http:
          pathPrefix: '/workflow'
      destination:
        service: workflow

The router seems to be created, but it shows up the error in consul-controller pod.

2021-06-26T21:16:34.035Z	INFO	controller.servicerouter	config entry not found in consul	{request: dev-arka/fmrouter}
2021-06-26T21:16:34.047Z	ERROR	controller.servicerouter	sync failed	{request: dev-arka/fmrouter, error: writing config entry to consul: Unexpected response code: 500 (rpc error making call: rpc error making call: discovery chain \fmrouter\ uses a protocol \tcp\ that does not permit advanced routing or splitting behavior)}

I am expecting the service router to intercept the requests to kubernetes service “fmrouter”. But this doesn’t seem to be working and looks like my requests are directly hitting the service without being proxied by the ServiceRouter.

Any ideas on what is wrong?

blake · June 27, 2021, 3:50am

Hi @parag,

That error is indicating that you need to create a service-defaults config entry for the fmrouter service which configures the service protocol to http. For example:

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: fmrouter
spec:
  protocol: http

You can find more information about this config entry here
Configuration Entry Kind: Service Defaults | Consul by HashiCorp.

parag · June 27, 2021, 6:49am

Hi @blake Thanks for your pointer.

However on doing above, I am getting the following error

ERROR	controller.servicerouter	sync failed	{"request": "dev-arka/fmrouter", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: rpc error making call: discovery chain \"fmrouter\" uses inconsistent protocols; service \"kc-wrap\" has \"tcp\" which is not \"http\")"}

Is this because my up-stream Kubernetes service (“kc-wrap”, a ClusterIP type service) has protocol as TCP? I don’t think it can be changed because Kubernetes service of ClusterIP type does not support HTTP as protocol.

My overall architecture is such that the external load balancer sends traffic to the Kubernetes Ingress which is then supposed to handover to the Kubernetes service (ClusterIP type). I was hoping the Consul Router would intercept the traffic before it reaches the Kubernetes service.

Any other suggestions that will help resolve the problem?

blake · June 27, 2021, 7:54am

Hi @parag,

When using a service-router to route to backend services, those backend services also need to have their protocol set to http.

If you only have a few HTTP-based services in your environment, its probably easiest to also create additional ServiceDefault configs with protocol set to http for each service.

If a majority of your services are HTTP-based, you can use the ProxyDefaults config entry to globally set the service protocol to http for all services.

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  config:
    protocol: http

If you add any TCP services, you can override this global default by creating a ServiceDefaults entry to change the protocol to TCP.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceDefaults
metadata:
  name: my_tcp_service
spec:
  protocol: tcp

I hope this helps. Let me know if you have any other questions.

parag · June 27, 2021, 8:56am

Hello @blake
Problem is that all my Kubernetes services uses TCP protocol. Kubernetes doesn’t support HTTP as protocol in a service configuration. I don’t yet know how can I make my Kubernetes service to listen on HTTP protocol.

While your suggestion helps with understanding that we can set the Service/Proxy Defaults which will determine the default behavior, but I am still clueless as to how it will help in the situation I am in.

Is it possible to make the ServiceRouter work with an upstream TCP Kubernetes service? The error seems to be suggesting it’s not.

ERROR	controller.servicerouter	sync failed	{"request": "dev-arka/fmrouter", "error": "writing config entry to consul: Unexpected response code: 500 (rpc error making call: rpc error making call: discovery chain \"fmrouter\" uses inconsistent protocols; service \"kc-wrap\" has \"tcp\" which is not \"http\")"}

Or is it possible to make the protocol of a Kubernetes service as HTTP?

Parag

blake · June 27, 2021, 9:42am

Hi @parag,

This configuration is related to how Consul sees services within the mesh. The protocol that is configured for the service is separate from the protocol associated with the Kubernetes Service object.

Have you tried creating a ServiceDefaults for the kc-wrap service, or globally changing the protocol using ProxyDefaults? Are you still receiving this error after creating either of those two config entries?

parag · June 27, 2021, 2:00pm

@blake
This really helps. The errors went away when I created ServiceDefault (with http protocol) for both fmrouter and kc-wrap services.

However my requests to fmrouter service still does not seem to be intercepted by the ServiceRouter. Requests are still hitting the fmrouter directly without being intercepted by the router.

Is there any other way I can debug to fix this?

lkysow · June 27, 2021, 5:02pm

Are you using the latest version of consul-helm? You can see which version you’re using by running helm list -f consul can you paste that output along with your values.yaml file and the yaml for your deloyments and services.

parag · June 28, 2021, 5:41am

@lkysow
I realized I was not accessing the service in consul way which is through the side car proxy

curl  http://localhost:1234/

and instead trying to access it directly as a Kubernetes service which is right now exposed to all interfaces. (All my services can be accessed from 0.0.0.0.)

So I configured my client with consul annotations like these

consul.hashicorp.com/connect-inject: "true"
consul.hashicorp.com/connect-service-upstreams: static-server:1234

And now I am getting the response from the static server.

curl  http://localhost:1234/
"hello world"

However in case of “/iam” route (as configured in Service Router), I am expecting the route to proxy to an upstream server (kc-wrap)

Below is my service router

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"consul.hashicorp.com/v1alpha1","kind":"ServiceRouter","metadata":{"annotations":{},"name":"static-server","namespace":"dev-arka"},"spec":{"routes":[{"destination":{"service":"kc-wrap"},"match":{"http":{"pathPrefix":"/iam"}}}]}}
  creationTimestamp: "2021-06-27T19:18:11Z"
  finalizers:
  - finalizers.consul.hashicorp.com
  generation: 5
  name: static-server
  namespace: dev-arka
  resourceVersion: "116984579"
  selfLink: /apis/consul.hashicorp.com/v1alpha1/namespaces/dev-arka/servicerouters/static-server
  uid: e3254dcd-f98c-442c-b91d-d835c01b8997
spec:
  routes:
  - destination:
      prefixRewrite: /
      service: kc-wrap
    match:
      http:
        pathPrefix: /iam
status:
  conditions:
  - lastTransitionTime: "2021-06-28T04:28:30Z"
    status: "True"
    type: Synced
  lastSyncedTime: "2021-06-28T04:28:30Z"

But I get the error (no healthy upstream) as below

curl  http://localhost:1234/iam
upstream connect error or disconnect/reset before headers. reset reason: connection failure/

P.S. I am able to access the Kubernetes kc-wrap service directly (without the side car proxy), but not via the router.

Where did I go wrong?

lkysow · June 28, 2021, 5:27pm

Hi, first, if you’re using the latest version of consul-helm then you can use the KubeDNS URL instead of the upstream annotation.

Second, is your kc-wrap service registered into Consul? Can you see it in the Consul UI? Its pods must also be connect injected.

parag · June 29, 2021, 4:39pm

Hello @lkysow I was earlier using the older release of consul. I updated to latest (0.32.0), and have made some further progress.

helm install consul hashicorp/consul --version 0.32.0 -f config.yaml

#config.yaml
global:
  name: consul
connectInject:
  enabled: true
  default: true
  metrics:
    defaultEnableMerging: true
controller:
  enabled: true
ui:
  enabled: true
  metrics_provider: prometheus
  metrics_proxy:
    base_url: "http://prometheus-server"
prometheus:
  enabled: true

The request from the ServiceRouter is reaching the destination as shown by the logs of the envoy-sidecar container of the destination.

[2021-06-29 15:43:22.477][13][debug][http] [source/common/http/conn_manager_impl.cc:261] [C15] new stream
[2021-06-29 15:43:22.477][13][debug][http] [source/common/http/conn_manager_impl.cc:882] [C15][S744548391255673329] request headers complete (end_stream=false):

':authority', 'localhost:1234'
':path', '/login'
':method', 'POST'
'user-agent', 'curl/7.52.1'
'accept', '*/*'
'content-type', 'application/json'
'content-length', '48'
'x-forwarded-proto', 'http'
'x-request-id', '4e82bcc1-8601-47b4-87fe-92626b766d57'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-envoy-original-path', '/iam/login'

[2021-06-29 15:43:22.477][13][debug][http] [source/common/http/filter_manager.cc:779] [C15][S744548391255673329] request end stream
[2021-06-29 15:43:22.477][13][debug][http] [source/common/http/filter_manager.cc:883] [C15][S744548391255673329] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2021-06-29 15:43:22.477][13][debug][http] [source/common/http/conn_manager_impl.cc:1469] [C15][S744548391255673329] encoding headers via codec (end_stream=false):

':status', '503'
'content-length', '91'
'content-type', 'text/plain'
'date', 'Tue, 29 Jun 2021 15:43:22 GMT'
'server', 'envoy'

The status in response received by the sidecar container is 503 which is a problem.

One more critical observation is this log that the destination container pod generates at startup.

Running on http://0.0.0.0:8181
{ Error: connect ECONNREFUSED 18.216.238.224:443
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '18.216.238.224',
  port: 443 }

The application runs on port 8181. I don’t know if the above error in application log is related to the side car proxies that get created due to connect-inject.

One more rather unrelated observation is in the consul-ui topology.

The expected flow is static-client → static-server → kc-wrap.
The additional kc-wrap on the left and static-client on the right seems to be misplaced. Any thing wrong in my configuration?

lkysow · June 29, 2021, 6:45pm

I think the 503 is because the Envoy proxy isn’t able to reach the application on its local port 8181. Can you confirm it’s listening on that port? The log you pasted looks like it’s trying to bind to 443.

Regarding the topology view, I’d need to see the annotations you have on each of the deployments.

parag · June 30, 2021, 10:03am

@lkysow
Indeed my upstream application did had some issue. When I replaced the problematic upstream application with another one, it started working like charm

The issue related to topology with all services showing both in upstream and downstream was due to the intention allowing all services both in source and destination. That too was fixed when I changed the intention.

Now the only issue is with some metrics not loading.

Following is the helm consul configuration used to set up consul with Kubernetes.

#helm consul configuration
global:
  name: consul
connectInject:
  enabled: true
  default: false
  metrics:
    defaultEnabled: true
    defaultEnableMerging: true
controller:
  enabled: true
ui:
  enabled: true
  metrics_provider: prometheus
  metrics_proxy:
    base_url: "http://prometheus-server"
prometheus:
  enabled: true

I see the following xhr requests failing in chrome inspector’s Network tab.

https://consul-ui-url/v1/health/service/static-server?dc=dc1&index=12363

https://consul-ui-url/v1/catalog/connect/static-server?dc=dc1&index=12363

https://consul-ui-url/v1/internal/ui/service-topology/static-server?dc=dc1&kind=&index=12365

https://consul-ui-url/v1/connect/intentions?dc=dc1&index=12013&filter=SourceName%20%3D%3D%20"static-server"%20or%20DestinationName%20%3D%3D%20"static-server"%20or%20SourceName%20%3D%3D%20"*"%20or%20DestinationName%20%3D%3D%20"*"

Can you help me with this last step in my wonderful journey with Consul

parag · July 1, 2021, 6:00pm

@blake @lkysow
Can you help me with the above issue with one of the consul ui metrics not loading?The XHR requests that seem to be going to fetch this data are failing with gateway timeout.

You can see the attached image above in which static-server metrics not loading.

Thanks in advance.
Parag

blake · July 1, 2021, 7:31pm

@parag, would you mind sharing the particular HXR error that you’re seeing in the browser’s logs? That would help with debugging this issue.

Thanks.

parag · July 1, 2021, 8:08pm

@blake
PFA the screen shot.

I can do a zoom/google meet session as well is needed.

Let me know.

Thanks
Parag

parag · July 2, 2021, 5:14pm

@blake @lkysow
Can you please help resolving the above issue with consul-ui?

Thanks
Parag

lkysow · July 6, 2021, 5:25pm

Hi Parag,
That looks like an issue with the load balancer in front of Consul. If you do a kubectl port-forward directly to one of the Consul server pods do you get the 504?

parag · July 6, 2021, 7:02pm

Tried direct port forward, but still facing the same issue. Only the above 4 mentioned APIs fail. However these four APIs are failing for all the services in consul mesh.

iftash · August 12, 2021, 2:59pm

Any update on this issue?