I have a deployment with 6 pods. All pods in the deployment are pointed at by one service. The pods sometimes send curl requests to their own service, which one of the pods in the deployment would respond to. This functionality works without a service mesh, as well as with Istio and Linkerd. However, when I attempt to run this using Consul, it fails.
I have the following pods:
kubectl get pods
NAME READY STATUS RESTARTS AGE
counter-5d5fdcb75c-ml2tx 2/2 Running 0 17h
counter-5d5fdcb75c-rs6cv 2/2 Running 0 17h
counter-5d5fdcb75c-sv692 2/2 Running 0 17h
counter-5d5fdcb75c-wkzw9 2/2 Running 0 17h
counter-5d5fdcb75c-xcqqp 2/2 Running 0 17h
counter-5d5fdcb75c-xhscb 2/2 Running 0 17h
request-generator-584977dc7f-9hsn4 2/2 Running 0 17h
and the following Services:
kubectl get services
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 25h
counter-service ClusterIP 10.96.74.63 <none> 80/TCP 21h
request-generator-service ClusterIP 10.96.112.70 <none> 80/TCP 25h
I can ssh into the pods and they can curl pods from another service, however, they cannot curl themselves. I even tried using their IP address but that would not work either.
SSH into the counter
$ kubectl exec --tty --stdin counter-5d5fdcb75c-ml2tx -- /bin/bash
Defaulted container "counter" out of: counter, consul-dataplane, consul-connect-inject-init (init)
root@counter-5d5fdcb75c-ml2tx:/ConsulTest# curl http://request-generator-service/home
I am the load generator. I use apache ab to send multiple requests to the counter.
root@counter-5d5fdcb75c-ml2tx:/ConsulTest# curl http://counter-service/home
curl: (52) Empty reply from server
root@counter-5d5fdcb75c-ml2tx:/ConsulTest# curl 10.96.112.70:80/home
I am the load generator. I use apache ab to send multiple requests to the micro-counter simulating a different number of concurrent users, requests per user, and micro-services to be pinged per request.
root@counter-5d5fdcb75c-ml2tx:/ConsulTest# curl 10.96.74.63:80/home
curl: (52) Empty reply from server
SSH into the generator
$ kubectl exec --tty --stdin request-generator-584977dc7f-9hsn4 -- /bin/bash
Defaulted container "request-generator" out of: request-generator, consul-dataplane, consul-connect-inject-init (init)
root@request-generator-584977dc7f-9hsn4:/ConsulTest# curl http://counter-service/home
I am the counter, when reached and provided with a value, I will increase my internal counter to that value.
root@request-generator-584977dc7f-9hsn4:/ConsulTest# curl http://request-generator-service/home
curl: (52) Empty reply from server
This is my manifest for the counter
apiVersion: v1
kind: Service
metadata:
name: counter-service
spec:
selector:
app: counter
ports:
- protocol: TCP
port: 80
targetPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: counter
labels:
app: counter
spec:
replicas: 6
selector:
matchLabels:
app: counter
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 3
maxUnavailable: 0
template:
metadata:
labels:
app: counter
annotations:
'consul.hashicorp.com/connect-inject': 'true'
spec:
containers:
- name: counter
image: counter:1.0
imagePullPolicy: Never
ports:
- containerPort: 5000
It seems like consul-injected pods can send requests to other services, but cannot send a request to their own service. Is there anyway to resolve this?