Tutorial - x509: certificate signed by unknown authority

Terraform Providers Google
1

Hi,
Following this tutorial, https://learn.hashicorp.com/tutorials/terraform/gke?in=terraform/kubernetes, I have deployed a GKE cluster in GCloud.

Now when I try to schedule a deployment following this link,

It fails with,
kubernetes_deployment.nginx: Creating…

Error: Failed to create deployment: Post "https://<ip>/apis/apps/v1/namespaces/default/deployments": x509: certificate signed by unknown authority

  on kubernetes.tf line 21, in resource "kubernetes_deployment" "nginx":
  21: resource "kubernetes_deployment" "nginx" {

My kubernetes.tf looks like this,

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

provider "kubernetes" {
  load_config_file = false

  host     = google_container_cluster.primary.endpoint
  username = var.gke_username
  password = var.gke_password

  client_certificate     = google_container_cluster.primary.master_auth.0.client_certificate
  client_key             = google_container_cluster.primary.master_auth.0.client_key
  cluster_ca_certificate = google_container_cluster.primary.master_auth.0.cluster_ca_certificate
}

resource "kubernetes_deployment" "nginx" {
  metadata {
    name = "scalable-nginx-example"
    labels = {
      App = "ScalableNginxExample"
    }
  }

  spec {
    replicas = 2
    selector {
      match_labels = {
        App = "ScalableNginxExample"
      }
    }
    template {
      metadata {
        labels = {
          App = "ScalableNginxExample"
        }
      }
      spec {
        container {
          image = "nginx:1.7.8"
          name  = "example"

          port {
            container_port = 80
          }

          resources {
            limits {
              cpu    = "0.5"
              memory = "512Mi"
            }
            requests {
              cpu    = "250m"
              memory = "50Mi"
            }
          }
        }
      }
    }
  }
}

I am using MacOS to run terraform. Any help is appreciated.

Thanks,
Arun

2

Welcome! How did you create the key and certificate in that config? And did you create any when deploying your GKE cluster? It looks like they aren’t signed by the same CA chain, as a guess.

3

Hi, Thank you for the reply, I haven’t created any certs or keys. I was just going through those tutorials.

Cluster has been created with these two disabled,
Client certificate
Binary Authorization

Cluster CA Cert has been created and that’s referenced in the tf state file.

TF state file looks like this,

"master_auth": [
              {
                "client_certificate": "",
                "client_certificate_config": [
                  {
                    "issue_client_certificate": false
                  }
                ],
                "client_key": "",
                "cluster_ca_certificate": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSL3ozQ3ZwcnJmNnhYMTlsVWJFM295cUJyUmhxbEErMmRlSDJFPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
                "password": "",
                "username": ""
              }
4

If a client certificate isn’t required, why are you including these parameters? Maybe try removing them and see what error message you get.

5

Hi,

I get the below error after removing, client_certificate, client_key and cluster_ca_certificate parameters

kubernetes_deployment.nginx: Still creating… [30s elapsed]
2020/12/13 16:45:23 [DEBUG] kubernetes_deployment.nginx: apply errored, but we’re indicating that via the Error pointer rather than returning it: Failed to create deployment: Post “http:///apis/apps/v1/namespaces/default/deployments”: dial tcp :80: i/o timeout

6

Sorry, my k8s experience is limited. I’ll see whether I can get this demo up and running tomorrow. Hopefully someone with more experience can chime in with help before then.

7

No worries at all, thank you so much.
I think by removing those certificates it’s trying to reach 80 which might not work.

I wonder if it’s somehow related to the Mac keychain not accepting the certificate that’s presented, when terraform reaches the cluster address.

8

Hi,
Issue was resolved by base64 decoding the cluster_ca_certificate file like this,

provider "kubernetes" {
  load_config_file = false

  host     = google_container_cluster.primary.endpoint
  username = var.gke_username
  password = var.gke_password

  client_certificate     = base64decode(google_container_cluster.primary.master_auth.0.client_certificate)
  client_key             = base64decode(google_container_cluster.primary.master_auth.0.client_key)
  cluster_ca_certificate = base64decode(google_container_cluster.primary.master_auth.0.cluster_ca_certificate)
}

But I now receive a new error saying,

Error: Failed to create deployment: deployments.apps is forbidden: User “system:anonymous” cannot create resource “deployments” in API group “apps” in the namespace “default”

I suppose I will raise a new ticket for this.

Thank you so much @jlj7 for looking at this and replying, much appreciated.

Thanks,
Arun

1 Like
9

I’m currently working on the same issue, and I can tell you why you are getting the “system:anonymous” message. As of K8s 1.19, basic authentication (ie, username and password) to the Kubernetes API has been disabled. Thus, your Kubernetes provider is not able to log into the cluster API and the cluster is therefore treating you as a non-user.

I have not been able to test this idea, but I think if you enable “Client certificate” in the container cluster config you should be able to use the “client_certificate” and “client_key” fields in the Kubernetes provider definition.

I have a legacy cluster that was upgraded to 1.19 while my terraform code is still using basic auth (long story) and I am currently trying to use the exec plugin to pull the credentials from gcloud. Fingers crossed.

10

Hi you need to add a cluster role bind, this error is origin from the rbac.
You can read about it here, Using RBAC Authorization | Kubernetes

This is the resource you need
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/cluster_role_binding

Consider for this exercise to give your self cluster admin role

Something like so

resource "kubernetes_cluster_role_binding" "sa-cluster-bind" {
  metadata {
    name = "cluster-admin-binding"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind = "ClusterRole"
    name = "cluster-admin"
  }
  subject {
    kind = "User"
    name = local.account_id["client_email"]
    api_group = "rbac.authorization.k8s.io"
  }
}

BTW, The kubernetes config should be like so

data "google_client_config" "current" {}
provider "kubernetes" {
  host = google_container_cluster.primary.endpoint
  cluster_ca_certificate = base64decode(google_container_cluster.primary.master_auth[0].cluster_ca_certificate, )
  token = data.google_client_config.current.access_token
}