Getting Issue in Deploy Single Consul Datacenter Across Multiple Kubernetes Clusters

Consul
, ,
1

Hi Team,

By referring to consul documentation https://developer.hashicorp.com/consul/docs/k8s/deployment-configurations/single-dc-multi-k8s
I am trying to Deploy Single Consul Datacenter Across Multiple Kubernetes Clusters. I have two k8 clusters, in one cluster I have installed consul server using helm and in other cluster I have installed consul connect using helm. Consul Connect on second cluster is joined to consul server on first cluster.

As given in documentation, I have deployed static-server service in first cluster where consul server is running and deployed static-client service in second cluster where the consul connect is running.
Now both the services static-server and static-client are registered with consul server and am able to view both the services in consul UI and in cli using consul catalog services.

But the issue is that static-client service in second cluster is unable to connect/communicate with static-server service in first cluster. If I deploy both services in same cluster then they are able to connect but when they are in different clusters they are unable to connect.

Output of curl command:

[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- curl localhost:1234
curl: (56) Recv failure: Connection reset by peer
command terminated with exit code 56

I am able to dig the static-server service from static client as well, but then am not understanding why they are unable to communicate. Below is output of dig command.

[root@devops consul]# kubectl exec -it static-client-5c6fcb95bb-xc4zx -c static-client -- bash
bash-5.1# dig @127.0.0.1 -p 8600 static-server.service.consul

; <<>> DiG 9.16.20 <<>> @127.0.0.1 -p 8600 static-server.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27185
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;static-server.service.consul.	IN	A

;; ANSWER SECTION:
static-server.service.consul. 0	IN	A	10.244.0.16

;; Query time: 2 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Thu Mar 30 13:49:41 UTC 2023
;; MSG SIZE  rcvd: 73

bash-5.1#

Below are my helm chart values.yaml files:

cluster1-values.yaml:

global:
  datacenter: dc1
  tls:
    enabled: true
    enableAutoEncrypt: true
  acls:
    manageSystemACLs: true
  gossipEncryption:
    secretName: consul-gossip-encryption-key
    secretKey: key
server:
  exposeService:
    enabled: true
    type: NodePort
    nodePort:
      ## all are random nodePorts and you can set your own
      http: 30010
      https: 30011
      serf: 30012
      rpc: 30013
      grpc: 30014
ui:
  service:
    type: NodePort

cluster2-values.yaml:

global:
  enabled: false
  datacenter: dc1
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: server-consul-bootstrap-acl-token
      secretKey: token
  tls:
    enabled: true
    caCert:
      secretName: server-consul-ca-cert
      secretKey: tls.crt
externalServers:
  enabled: true
  hosts: ["172.18.0.2"]
  httpsPort: 31306
  grpcPort: 30014
  tlsServerName: server.dc1.consul
  k8sAuthMethodHost: https://172.18.0.3:6443
connectInject:
  enabled: true

Below are the yaml files of both services:

static-server.yaml

---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceIntentions
metadata:
  name: static-server
spec:
  destination:
    name: static-server
  sources:
    - name: static-client
      action: allow
---
apiVersion: v1
kind: Service
metadata:
  name: static-server
spec:
  type: ClusterIP
  selector:
    app: static-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http
      serviceAccountName: static-server

static-client.yaml

apiVersion: v1
kind: Service
metadata:
  name: static-client
spec:
  selector:
    app: static-client
  ports:
    - port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-client
  template:
    metadata:
      name: static-client
      labels:
        app: static-client
      annotations:
        "consul.hashicorp.com/connect-inject": "true"
        "consul.hashicorp.com/connect-service-upstreams": "static-server:1234"
    spec:
      containers:
        - name: static-client
          image: praqma/network-multitool
          command: [ "/bin/sh", "-c", "--" ]
          args: [ "while true; do sleep 30; done;" ]
      serviceAccountName: static-client

Kindly can someone look into the configuration and help me out in resolving this issue.

2

Hey, Can someone look into this and help me