By referring to consul documentation https://developer.hashicorp.com/consul/docs/k8s/deployment-configurations/single-dc-multi-k8s
I am trying to Deploy Single Consul Datacenter Across Multiple Kubernetes Clusters. I have two k8 clusters, in one cluster I have installed consul server using helm and in other cluster I have installed consul connect using helm. Consul Connect on second cluster is joined to consul server on first cluster.
As given in documentation, I have deployed static-server service in first cluster where consul server is running and deployed static-client service in second cluster where the consul connect is running.
Now both the services static-server and static-client are registered with consul server and am able to view both the services in consul UI and in cli using consul catalog services.
But the issue is that static-client service in second cluster is unable to connect/communicate with static-server service in first cluster. If I deploy both services in same cluster then they are able to connect but when they are in different clusters they are unable to connect.
Output of curl command:
[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- curl localhost:1234 curl: (56) Recv failure: Connection reset by peer command terminated with exit code 56
I am able to dig the static-server service from static client as well, but then am not understanding why they are unable to communicate. Below is output of dig command.
[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- bash bash-5.1# dig @127.0.0.1 -p 8600 static-server.service.consul ; <<>> DiG 9.16.20 <<>> @127.0.0.1 -p 8600 static-server.service.consul ; (1 server found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27185 ;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; WARNING: recursion requested but not available ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;static-server.service.consul. IN A ;; ANSWER SECTION: static-server.service.consul. 0 IN A 10.244.0.16 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#8600(127.0.0.1) ;; WHEN: Thu Mar 30 13:49:41 UTC 2023 ;; MSG SIZE rcvd: 73 bash-5.1#
Below are my helm chart values.yaml files:
global: datacenter: dc1 tls: enabled: true enableAutoEncrypt: true acls: manageSystemACLs: true gossipEncryption: secretName: consul-gossip-encryption-key secretKey: key server: exposeService: enabled: true type: NodePort nodePort: ## all are random nodePorts and you can set your own http: 30010 https: 30011 serf: 30012 rpc: 30013 grpc: 30014 ui: service: type: NodePort
global: enabled: false datacenter: dc1 acls: manageSystemACLs: true bootstrapToken: secretName: server-consul-bootstrap-acl-token secretKey: token tls: enabled: true caCert: secretName: server-consul-ca-cert secretKey: tls.crt externalServers: enabled: true hosts: ["172.18.0.2"] httpsPort: 31306 grpcPort: 30014 tlsServerName: server.dc1.consul k8sAuthMethodHost: https://172.18.0.3:6443 connectInject: enabled: true
Below are the yaml files of both services:
--- apiVersion: consul.hashicorp.com/v1alpha1 kind: ServiceIntentions metadata: name: static-server spec: destination: name: static-server sources: - name: static-client action: allow --- apiVersion: v1 kind: Service metadata: name: static-server spec: type: ClusterIP selector: app: static-server ports: - protocol: TCP port: 80 targetPort: 8080 --- apiVersion: v1 kind: ServiceAccount metadata: name: static-server --- apiVersion: apps/v1 kind: Deployment metadata: name: static-server spec: replicas: 1 selector: matchLabels: app: static-server template: metadata: name: static-server labels: app: static-server annotations: "consul.hashicorp.com/connect-inject": "true" spec: containers: - name: static-server image: hashicorp/http-echo:latest args: - -text="hello world" - -listen=:8080 ports: - containerPort: 8080 name: http serviceAccountName: static-server
apiVersion: v1 kind: Service metadata: name: static-client spec: selector: app: static-client ports: - port: 80 --- apiVersion: v1 kind: ServiceAccount metadata: name: static-client --- apiVersion: apps/v1 kind: Deployment metadata: name: static-client spec: replicas: 1 selector: matchLabels: app: static-client template: metadata: name: static-client labels: app: static-client annotations: "consul.hashicorp.com/connect-inject": "true" "consul.hashicorp.com/connect-service-upstreams": "static-server:1234" spec: containers: - name: static-client image: praqma/network-multitool command: [ "/bin/sh", "-c", "--" ] args: [ "while true; do sleep 30; done;" ] serviceAccountName: static-client
Kindly can someone look into the configuration and help me out in resolving this issue.