How to install and run tcpdump on the envoy-sidecar

I have installed consul via helm chart 0.40.0 into our TKGi (VMware) env. and I need to capture a tcpdump from a envoy-sidecar proxy container to demonstrate tls encryption to the upstream/downstream proxy.

The K8s version is 1.22.2
The installed proxy image - envoyproxy/envoy-alpine:v1.20.1:

  1. does not include tcpdump or apt-get.
  2. the running user in the side-car is unknown.
    / id uid=5995 gid=5995 / whoami
    whoami: unknown uid 5995
    / $ su -
    su: Cannot determine your user name.

Can you help me figure out how to install and use tcpdump to capture upstream data?

The description of containers in the running pod:

Init Containers:
copy-consul-bin:
Container ID: containerd://203b4dd5390a10f5dd1781edede58863563979b26be94d45e6866f3d15d3889c
Image: hashicorp/consul:1.11.2
Image ID: docker.io/hashicorp/consul@sha256:8e06a85e185ca2f2eeb65e91ef67e4d0c26aaa70f9a5da9619cfab5f3d6cb394
Port:
Host Port:
Command:
/bin/sh
-ec
cp /bin/consul /consul/connect-inject/consul
State: Terminated
Reason: Completed
Exit Code: 0
Started: Thu, 10 Feb 2022 20:38:18 +0000
Finished: Thu, 10 Feb 2022 20:38:20 +0000
Ready: True
Restart Count: 0
Limits:
cpu: 50m
memory: 150Mi
Requests:
cpu: 50m
memory: 25Mi
Environment:
Mounts:
/consul/connect-inject from consul-connect-inject-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6g8rk (ro)
consul-connect-inject-init:
Container ID: containerd://1c12ba707e2896a36f6fbfd471ca3a65b7d976a54a4aa088931179badd2ae7fb
Image: hashicorp/consul-k8s-control-plane:0.40.0
Image ID: docker.io/hashicorp/consul-k8s-control-plane@sha256:cf2a96b024d20088e9aad6abef6cac2a9c6c19466eee9d82a3b6d03f79903e0f
Port:
Host Port:
Command:
/bin/sh
-ec

  export CONSUL_HTTP_ADDR="${HOST_IP}:8500"
  export CONSUL_GRPC_ADDR="${HOST_IP}:8502"
  consul-k8s-control-plane connect-init -pod-name=${POD_NAME} -pod-namespace=${POD_NAMESPACE} \

  # Generate the envoy bootstrap code
  /consul/connect-inject/consul connect envoy \
    -proxy-id="$(cat /consul/connect-inject/proxyid)" \
    -bootstrap > /consul/connect-inject/envoy-bootstrap.yaml

  # Apply traffic redirection rules.
  /consul/connect-inject/consul connect redirect-traffic \
    -proxy-id="$(cat /consul/connect-inject/proxyid)" \
    -proxy-uid=5995
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Thu, 10 Feb 2022 20:38:21 +0000
  Finished:     Thu, 10 Feb 2022 20:38:27 +0000
Ready:          True
Restart Count:  0
Limits:
  cpu:     50m
  memory:  150Mi
Requests:
  cpu:     50m
  memory:  25Mi
Environment:
  HOST_IP:         (v1:status.hostIP)
  POD_IP:          (v1:status.podIP)
  POD_NAME:       util-sidecar-secure-6c46d9c89c-4frdc (v1:metadata.name)
  POD_NAMESPACE:  kimtest (v1:metadata.namespace)
Mounts:
  /consul/connect-inject from consul-connect-inject-data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6g8rk (ro)

Containers:
util:
Container ID: containerd://807c905a795870a71b8b49d40861f1b96cf7725f5e984e50a39931ea23fa162a
Image: hashicorp/consul:1.11.2
Image ID: docker.io/hashicorp/consul@sha256:8e06a85e185ca2f2eeb65e91ef67e4d0c26aaa70f9a5da9619cfab5f3d6cb394
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/sleep
infinity
State: Running
Started: Thu, 10 Feb 2022 20:38:28 +0000
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6g8rk (ro)
envoy-sidecar:
Container ID: containerd://5e3bdee087c2a7a498468011a12b41a9dbace33f3bbf9b2494eaf79fe568d36f
Image: envoyproxy/envoy-alpine:v1.20.1
Image ID: docker.io/envoyproxy/envoy-alpine@sha256:589805bf0d51dbde5b9635995a2da2047982221605ed7c97a5764d1a84510e9f
Port:
Host Port:
Command:
envoy
–config-path
/consul/connect-inject/envoy-bootstrap.yaml
–log-level
debug
State: Running
Started: Thu, 10 Feb 2022 20:38:28 +0000
Ready: True
Restart Count: 0
Environment:
HOST_IP: (v1:status.hostIP)
Mounts:
/consul/connect-inject from consul-connect-inject-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6g8rk (ro)

  1. Are you able to use ephemeral debug containers (Debug Running Pods | Kubernetes)?
  2. Have you tried kubectl exec <pod-name> -c util -- sh? This will get you into the util container from which you should be able to run apk install tcpdump.

I’ve tried both. I am looking to capture and demonstrate encrypted data pod-to-pod. So,

  1. I have captured on the interface of the util container, but it sends unencrypted to the envoy sidecar.
  2. Ephemeral containers are disabled in this TKGi cluster environment. I see this when I run kubectl debug: error:
    “ephemeral containers are disabled for this cluster (error from server: “the server could not find the requested resource”).”
  3. Most unusual, In this circumstance I would tap into the container interface via shell access to the cluster node but, unexpectedly, the consul installation seems to have moved/deleted/ or otherwise obscured the docker binary. That binary is usually found on the node at /var/vcap/packages/docker but it’s missing only on the nodes of this cluster that consul is deployed in.

Can you help me,
a. determine how to deploy consul so that it does not change the docker binary on the cluster nodes?
b. understand high level how to change the envoy-sidecar deployment so that it runs as root, or otherwise allow me to install tcpdump after the pod has been deployed.
c. guide me to a high-level instruction on building my own custom envoy image that includes the necessary packages including tcpdmp.

Hi, re 1) the communication between app and sidecar is unencrypted, it’s just the communication between sidecars that’s encrypted, e.g. when foo talks to bar:

foo => foo-sidecar (unencrypted)
foo-sidecar => bar-sidecar (encrypted)
bar-sidecar => bar (unencrypted)

So you should be able to use tcpdump to verify outgoing traffic from foo-sidecar is encrypted or incoming traffic on bar-sidecar is encrypted. The incoming traffic is on port 20000.

  1. I don’t know how this could be the Consul installation since it doesn’t modify anything on the underlying host.

Running this on the source pod to look at outgoing packets to port 20000 worked for me:

 tcpdump tcp and src $(hostname -i) and dst port 20000 -A

(The hostname -i command is just to get the local IP)

Then I did a curl to the remote service and got this dump:

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
18:31:32.255014 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [S], seq 2489890047, win 26883, options [mss 8961,sackOK,TS val 3660993722 ecr 0,nop,wscale 7], length 0
E..<tQ@...3n......>..
N .h........i..*....#....
.6T.........
18:31:32.255794 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [.], ack 2263853331, win 211, options [nop,nop,TS val 3660993723 ecr 3981988357], length 0
E..4tR@...3u......>..
N .h...........".....
.6T..XR.
18:31:32.255843 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [P.], seq 0:980, ack 1, win 211, options [nop,nop,TS val 3660993723 ecr 3981988357], length 980
E...tS@.../.......>..
N .h.................
.6T..XR............bUt.F....Y.......>Gu....L....!.. .O`..j..D.e[...1...].WT..Yk>..ae...+.../...,.0...v...M.K..Hhttpbin.default.dc1.internal.90abbee2-c7d0-431b-0921-8be8fe525971.consul..........
...............#...	....\..'p.b.......".t8.j	.p..v..|...l.e...l.J.T....l.......g6....t.........ujX*p.lO......E.....~...Q..bo...N.n.R:..rG..b..uS..].6. ....... Y=.%....1.3=|yq4R.z.f.a...J..0=.c..h].Q..~ ..kl..q."..y.iJ.m~+.............W..Z.u....[W...
.K.>.G~-..*..e..W..P.......%.$..B$.x2>'.qW.`..:...z...Y.....0.:k...O..6p.X.G..I.r.E...W...1..rX.	%t...a.z2[x....j..>.....0.%.Il.2]..+..;,......q.{...
....).*dt)....F>...Gt.^..<h....Xz....E.......W=o.,<...E.|".Ar..V.H......[v(...6|TS...=>.......Q..=t.>M...7..!.....KJ....m.#...qsW...p..%Q.......M.[.D...=...........p|..?.RL..Q.,)x^^..	.5.7..U..^.ye....`0...p.1......IFv..m...P...1..T.......,.g'.b........(z8.o...,.....3b..._.....)..i.Dq(..E...o.t`..e.........e.DL... ."N.[.]%.bX.a?..IHz....:....'H...rnPs..B.u.....'.k~
..........................
18:31:32.256845 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [.], ack 148, win 219, options [nop,nop,TS val 3660993724 ecr 3981988358], length 0
E..4tT@...3s......>..
N .h...........".....
.6T..XR.
18:31:32.256910 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [P.], seq 980:1031, ack 148, win 219, options [nop,nop,TS val 3660993724 ecr 3981988358], length 51
E..gtU@...3?......>..
N .h...........U.....
.6T..XR...........(..........Z...^O#`|....Ymm.Py...N.....p.
18:31:32.256995 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [P.], seq 1031:1138, ack 148, win 219, options [nop,nop,TS val 3660993724 ecr 3981988358], length 107
E...tV@...3.......>..
N .h.................
.6T..XR.....f........k.....\M/.9...[.....$.....?.,...@..................=.Nl.ny...x.m.C...~R.B.g...R:..L,<	...cb.J}
18:31:32.259731 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [P.], seq 1138:1169, ack 585, win 227, options [nop,nop,TS val 3660993727 ecr 3981988361], length 31
E..StW@...3Q......>..
N .h.r...[.....A.....
.6T..XR	...............q=..s...f....(qJ
18:31:32.260759 IP static-client-66d777b99f-9xff9.47370 > 192-168-62-223.httpbin.default.svc.cluster.local.20000: Flags [F.], seq 1169, ack 617, win 227, options [nop,nop,TS val 3660993728 ecr 3981988362], length 0
E..4tX@...3o......>..
N .h.....{.....".....
.6T..XR
1 Like