Hello Team,
Thank yo so much for the awesome Consul. I have a k8s cluster where i am running consul cluster and i have another kubernetes cluster that i have to run our services. I have set up agents on the workload cluster able to WAN join the server and the cluster. How to set up connect inject in this case . On externalServers.hosts, i have given the k8s-autojoin config for the k8s cluster hosting the consul servers. What will be the k8sAuthMethodHost should have? LocalCluster’s kube-api server address or the ones that is hosting consul servers. Also advice on the certificates that i need to use in this case . i have one for the local cluster and one for the k8s cluster hosting the server instances …
The k8sAuthMethodHost should be the workload cluster’s API server address. The way it works is when the workload cluster’s connect pods start, they send their service account token to the consul servers. The consul servers then make an API call to the kubernetes api to check if this service account token is valid. Since the service account token came from the workload cluster, the k8sAuthMethodHost needs to be the API server of the workload cluster.
Also worth noting in terms of terminology, WAN join is only between multiple datacenters. Since you’re connecting servers and clients together that is considered the LAN still.
Hello Ikysow,
Thank you so much for the response. Configured Consul as per your advise and managed to configure consul servers on one kubernetes cluster and the clients on other cluster . But on connect inject, getting the below error.
Blockquote
Error logging in: Unexpected response code: 500 (rpc error making call: rpc error making call: Post “https://kubernetes.default.svc/apis/authentication.k8s.io/v1/tokenreviews”: x509: certificate signed by unknown authority (possibly because of “crypto/rsa: verification error” while trying to verify candidate authority certificate “kube-ca”))
Blockquote
The way i configured is run the helm chart on the first kubernetes cluster with connect and inject enabled , and once everything comes up, remove the consul agent daemon set , injector and mesh gateway, expose the consul servers port(https, grpc, serfLan) to host port export the gossip token , bootstrap token and consul ca certs an the second cluster, enabled external server and k8s cloud auto join config and the k8sAuthHost name owith the kube api server server name of the second cluster on helm values and brought it up. Did some modification on the acl-init job to mount the kubeconfig to connect to the consul servers running on the first cluster. Please advise if i am doing it right … Apologies if i am wrong …
Hi, it looks like that setting didn’t get passed since it’s still using the default Kubernetes hostname. Are you using the latest versions of consul-helm and consul-k8s?
That setting gets passed into the server-acl-init job via the flag -inject-auth-method-host.
Hello Ikysow,
Thank you for your response . Actually we are using rancher to provision K8s cluster and the kubernetes api endpoint will the proxy endpoint generated by rancher … Something like https://icp-mgmt-test-rnb.intra.com/k8s/clusters/c-58k5j This endpoint has a certificate signed by a different signer and the actual kubernetes api endpoint cert is signed by Kube ca. I tried with the above url and it gave me the same error x509: certificate signed by unknown authority
But one thing i noticed is that when i install the full helm chart on the cluster , the connect inject is working and the auth-method created was having the kubeapi server service url and that i used the same value as k8sauthMethodHost on the helm chart. Kindly advise whether my understanding is correct …
Ah right, because the CA cert we use comes from the service account of the cluster that the servers are running in. I think the unfortunate answer then is that we don’t support this architecture. The Consul servers must run in the same cluster as the workloads.
One option you might have is to manually change the auth method (https://www.consul.io/docs/acl/auth-methods/kubernetes) after the server-acl-init job runs and change the CACert and ServiceAccountJWT to a service account in your workload cluster.