Consul federation in K8s: x509: certificate signed by unknown authority

Consul
1

Hello, I’m trying to setup consul federation with ACL enabled on two kubernetes clusters managed by Rancher.

I’m getting following error in consul-connect-injector pod and in consul-mesh-gateway init container:
consul-server-connection-manager: connection error: error="rpc error: code = Unauthenticated desc = Post \"https://kubernetes.default.svc/apis/authentication.k8s.io/v1/tokenreviews\": x509: certificate signed by unknown authority"

Consul was installed by helm with following values:
DC1

connectInject:
  enabled: true
global:
  enabled: true
  name: consul
  domain: consul
  acls:
    createReplicationToken: true
    manageSystemACLs: true
  datacenter: dc1
  federation:
    createFederationSecret: true
    enabled: true
  gossipEncryption:
    autoGenerate: true
  tls:
    enabled: true
meshGateway:
  enabled: true

DC2

connectInject:
  enabled: true
global:
  enabled: true
  name: consul
  domain: consul
  acls:
    createReplicationToken: false
    manageSystemACLs: true
    replicationToken:
      secretKey: replicationToken
      secretName: consul-federation
  datacenter: dc2
  federation:
    createFederationSecret: false
    enabled: true
    k8sAuthMethodHost: https://kubernetes.default.svc
    primaryDatacenter: dc1
  gossipEncryption:
    autoGenerate: false
    secretKey: gossipEncryptionKey
    secretName: consul-federation
  tls:
    caCert:
      secretKey: caCert
      secretName: consul-federation
    caKey:
      secretKey: caKey
      secretName: consul-federation
    enabled: true
meshGateway:
  enabled: true
server:
  extraVolumes:
  - items:
    - key: serverConfigJSON
      path: config.json
    load: true
    name: consul-federation
    type: secret

I have used official documentation and tutorials to setup this.

Can anyone help to solve this issue?

2

Hi @alina.freydina,

Welcome to the HashiCorp Forums!

I think you are getting this error due to the federation.k8sAuthMethodHost you configured in the DC2 not correctly resolving to the DC2 K8S API from the DC1 cluster. DC1 should be able to hit the DC2 K8S API URL for token validation. (ref: WAN Federation Through Mesh Gateways - Multiple Kubernetes Clusters | Consul | HashiCorp Developer)

kubernetes.default.svc would resolve to the K8S API URL of the same cluster.

I recommend using an FQDN that resolves to DC2 K8S API URL from DC1 for federation.k8sAuthMethodHost and see if it works.

To test this, instead of a re-installation, you can try manually modifying the Consul Auth-Method in DC1 that points to DC2 with a right URL.

I hope this helps.

3

Hi Ranjandas, yes, you are right. It works now.
Thank you very much