Hi Team,
By referring to consul documentation https://developer.hashicorp.com/consul/docs/k8s/deployment-configurations/single-dc-multi-k8s
I am trying to Deploy Single Consul Datacenter Across Multiple Kubernetes Clusters. I have two k8 clusters, in one cluster I have installed consul server using helm and in other cluster I have installed consul connect using helm. Consul Connect on second cluster is joined to consul server on first cluster.
As given in documentation, I have deployed static-server service in first cluster where consul server is running and deployed static-client service in second cluster where the consul connect is running.
Now both the services static-server and static-client are registered with consul server and am able to view both the services in consul UI and in cli using consul catalog services.
But the issue is that static-client service in second cluster is unable to connect/communicate with static-server service in first cluster. If I deploy both services in same cluster then they are able to connect but when they are in different clusters they are unable to connect.
Output of curl command:
[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- curl localhost:1234
curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56
I am able to dig the static-server service from static client as well, but then am not understanding why they are unable to communicate. Below is output of dig command.
[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- bash
bash-5.1# dig @127.0.0.1 -p 8600 static-server.service.consul
; <<>> DiG 9.16.20 <<>> @127.0.0.1 -p 8600 static-server.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27185
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;static-server.service.consul. IN A
;; ANSWER SECTION:
static-server.service.consul. 0 IN A 10.244.0.16
;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Mar 30 13:49:41 UTC 2023
;; MSG SIZE rcvd: 73
bash-5.1#
Below are my helm chart values.yaml files:
cluster1-values.yaml:
global:
datacenter: dc1
tls:
enabled: true
enableAutoEncrypt: true
acls:
manageSystemACLs: true
gossipEncryption:
secretName: consul-gossip-encryption-key
secretKey: key
server:
exposeService:
enabled: true
type: NodePort
nodePort:
## all are random nodePorts and you can set your own
http: 30010
https: 30011
serf: 30012
rpc: 30013
grpc: 30014
ui:
service:
type: NodePort
cluster2-values.yaml:
global:
enabled: false
datacenter: dc1
acls:
manageSystemACLs: true
bootstrapToken:
secretName: server-consul-bootstrap-acl-token
secretKey: token
tls:
enabled: true
caCert:
secretName: server-consul-ca-cert
secretKey: tls.crt
externalServers:
enabled: true
hosts: ["172.18.0.2"]
httpsPort: 31306
grpcPort: 30014
tlsServerName: server.dc1.consul
k8sAuthMethodHost: https://172.18.0.3:6443
connectInject:
enabled: true
Below are the yaml files of both services:
static-server.yaml
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
name: static-server
spec:
destination:
name: static-server
sources:
- name: static-client
action: allow
---
apiVersion: v1
kind: Service
metadata:
name: static-server
spec:
type: ClusterIP
selector:
app: static-server
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: static-server
spec:
replicas: 1
selector:
matchLabels:
app: static-server
template:
metadata:
name: static-server
labels:
app: static-server
annotations:
"consul.hashicorp.com/connect-inject": "true"
spec:
containers:
- name: static-server
image: hashicorp/http-echo:latest
args:
- -text="hello world"
- -listen=:8080
ports:
- containerPort: 8080
name: http
serviceAccountName: static-server
static-client.yaml
apiVersion: v1
kind: Service
metadata:
name: static-client
spec:
selector:
app: static-client
ports:
- port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: static-client
spec:
replicas: 1
selector:
matchLabels:
app: static-client
template:
metadata:
name: static-client
labels:
app: static-client
annotations:
"consul.hashicorp.com/connect-inject": "true"
"consul.hashicorp.com/connect-service-upstreams": "static-server:1234"
spec:
containers:
- name: static-client
image: praqma/network-multitool
command: [ "/bin/sh", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
serviceAccountName: static-client
Kindly can someone look into the configuration and help me out in resolving this issue.