Weird behavior of the service mesh

Consul
, , ,
1

Hello, I’m currently doing some basic connectivity testing of the consul mesh v1.13.2, before we start the implementation in our project. My test scenario is the following:

scenario

Given I have the following services running:

  • service-a (ubuntu, meshed)
  • service-b (ubuntu, not meshed)
  • service-c (hashicorp/http-echo, meshed)
  • service-d (hashicorp/http-echo/ not meshed)

Note:

  • all meshed services are running transparent proxy
  • dev ns has the following metadata:
apiVersion: v1
kind: Namespace
metadata:
  labels:
    consul.hashicorp.com/connect-inject: enabled
    consul.hashicorp.com/transparent-proxy: "true"
    kubernetes.io/metadata.name: dev
  name: dev

The following deployments & services are present in the dev namespace:

k -n dev get pods
NAME                         READY   STATUS    RESTARTS   AGE
service-a-6898c564bc-55lbd   2/2     Running   0          17h
service-b-b6c4679d4-pq595    1/1     Running   0          17h
service-c-859d9d8678-jjm8f   2/2     Running   0          17h
service-c-859d9d8678-mz8nh   2/2     Running   0          17h
service-d-7f4494b8f6-ddjd2   1/1     Running   0          20h
service-d-7f4494b8f6-tkp5k   1/1     Running   0          20h
❯ k -n dev get services
NAME            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service-a-svc   ClusterIP   10.0.244.34    <none>        5678/TCP   2d19h
service-b-svc   ClusterIP   10.0.14.106    <none>        5678/TCP   2d21h
service-c-svc   ClusterIP   10.0.176.237   <none>        5678/TCP   2d21h
service-d-svc   ClusterIP   10.0.210.43    <none>        5678/TCP   20h

And I have the following service intentions defined:

❯ k -n consul exec -it consul-server-0 consul -- consul intention match -source service-a-svc
* => * (allow)

When I try to call D from B (both outside of the mesh):

❯ k -n dev exec -it service-b-b6c4679d4-pq595 -- bash
root@service-b-b6c4679d4-pq595:/# curl http://service-d-svc
Action: Deny. Reason: No rule matched. Proceeding with default action
root@service-b-b6c4679d4-pq595:/# curl http://service-d-svc.dev.svc.cluster.local
Action: Deny. Reason: No rule matched. Proceeding with default action.

Then:

  • I’m getting blocked, while calling D from B, despite the fact that * => * (allow) is in effect
  • Service intentions do not appear to have an effect, in all other cases, when setting them via consul UI
  • Service intentions do not appear to have an effect in all other cases, when setting them via kubectl as CRDs (dev namespace, same as the services)

Can anyone explain those phenomenons to me? It seems as if I’ve missed something simple.

2

Hi @mmisztal1980,

Welcome to the HashiCorp Forums!

Considering that services D and B are not part of the Service Mesh, the intentions won’t affect them. A quick search of your error messages shows that those messages are usually from Azure Firewall.

Are you running on AKS? Please verify our Network Policies to see why the connectivity is blocked.

Your k get pods also shows that Service B and D have only 1 container, which again shows that they are not part of mesh (no sidecar proxies).

I hope this helps.

3

You are correct, we are running AKS. We need to consult with our networking team, this looks like a potential root cause.

Thank you, we’ll let you know once we know for certain

1 Like