Checking How Consul Sidecar works [Kubernetes + Consul]

Consul
, ,
1

Dear all,

I have so far connected a K8S cluster with an external Consul Server. Also, I have registered 02 pods in K8s in Consul using connect-inject=true flag. Now (after adding the annotation to both pods), I am able to curl to the service name as below;

k exec -it pod/multitool-pod -c network-multitool -- curl nginx-service
Hello World! Response from Kubernetes! >> response

However, I cannot curl directly to the IP of the k8s-nginx pod

k exec -it pod/multitool-pod -c network-multitool -- curl 30.0.1.86
curl: (52) Empty reply from server
command terminated with exit code 52

I see that we can now only use the service name instead of the IP due to the way Consul sidecar works. But, I don’t fully understand why it happens?

I have few issues that I am unable to understand so far.

  1. I doubt that I am not even using consul to access the nginx pod because I am accessing it using the internal kubernetes service name. I also tried checking the below logs and there were no logs generated when I curl the service name. So I would like to know how can I access the services through Consul.

Checked logs:
kubectl logs -f pod/consul-consul-connect-injector-7f5c9f4f7-rrmz7 -n consul
kubectl logs -f pod/k8s-nginx-68d85bb657-b4rrs -c consul-dataplane
kubectl logs -f pod/multitool-pod -c consul-dataplane

  1. I came across consul proxy and upstream components. As I believe, I am running Transparent Proxy mode. I am using Envoy (Not the Consul built-in Proxy) because I can see consul-dataplane container inside these pods. Also, for upstream ( consul.hashicorp.com/connect-service-upstreams) it says that When transparent proxy is enabled, this annotation is optional.

Below are my configurations;

I have nginx-deployment, nginx service in Kubernetes. In nginx-deployment I have connect-inject flag set to true. In nginx service I have service-sync flag set to true.

Kubernetes consul-values.yaml

global:
  enabled: false
  tls:
    enabled: false
externalServers:
  enabled: true
  hosts: ["192.168.60.10"]
  httpsPort: 8500
  k8sAuthMethodHost: "https://192.168.50.10:6443"
server:
  enabled: false
syncCatalog:
  enabled: true
  toConsul: false
  toK8S: false

Consul Server => /etc/consul.d/consul.hcl

data_dir = "/opt/consul"
client_addr = "0.0.0.0"

ui_config{
  enabled = true
}

server = true
advertise_addr = "192.168.60.10"
bootstrap_expect=1
retry_join = ["192.168.60.10"]

ports {
 grpc = 8502 
}

connect {
 enabled = true
}

config_entries {
 bootstrap = [
  {
    kind = "proxy-defaults"
    name = "global"
    config {
      protocol = "http"
    }
  }
 ]
}

Here (in /etc/consul.d/consul.hcl) I have below code.

config_entries {
 bootstrap = [
  {
    kind = "proxy-defaults"
    name = "global"
    config {
      protocol = "http"
    }
  }
 ]
}

However, I tried to curl to service name even after removing this code block, which still works. So I guess this block doesn’t have any impact. So I removed it.

Consul UI

image

image
image894Ă—248 10.8 KB

image
image1508Ă—809 41.8 KB

Could someone kindly advice whether am I missing something here please.

Thank you!

2

Hi @harsh.lif3,

I see that we can now only use the service name instead of the IP due to the way Consul sidecar works. But, I don’t fully understand why it happens?

Ref: Transparent proxy overview | Consul | HashiCorp Developer

  1. I doubt that I am not even using consul to access the nginx pod because I am accessing it using the internal kubernetes service name. I also tried checking the below logs and there were no logs generated when I curl the service name. So I would like to know how can I access the services through Consul.

I think the above documentation would give some insights into how this is working. Yes, you are using the Kubernetes Service Name to access the service, but the requests are flowing through Consul/Envoy.

I would highly recommend you go through the following resources to understand how Consul Service Mesh works:

Although the material above doesn’t include transparent proxy, understanding it would make understanding transparent proxy easy.

Regarding Logging, by default, the sidecar logs are at INFO level, you will have to bump up the log-level to DEBUG to see the envoy operational logs (which includes requests and responses). Alternatively, you can enable Envoy Access Logging Proxy defaults configuration entry reference | Consul | HashiCorp Developer which would give traditional access logging which will be much easier to understand.

ref: Debug service mesh events and errors with Consul proxy access logs | Consul | HashiCorp Developer

  1. I came across consul proxy and upstream components. As I believe, I am running Transparent Proxy mode. I am using Envoy (Not the Consul built-in Proxy) because I can see consul-dataplane container inside these pods. Also, for upstream ( consul.hashicorp.com/connect-service-upstreams) it says that When transparent proxy is enabled, this annotation is optional.

You will understand this better if you go through the material I linked about (Life of a Packet Through Consul Service Mesh).

However, I tried to curl to service name even after removing this code block, which still works. So I guess this block doesn’t have any impact. So I removed it.

This config-entry makes a huge difference in that all the sidecar gets re-configured when you switch the service protocol from http to tcp. You will be able to take advantage of service mesh advanced L7 features only if the protocol is set to http (either in proxy-defaults or per service level using service-defaults).

I hope this helps.

1 Like