Kubernetes Auth method CA set by Terraform doesn't work

Terraform Providers
1

Hi all!

Setting up Kubernetes and Vault with Terraform I have a confusing issue. Can’t really tell if this is TF or Vault problem, both or PEBKAC.

The CA set by TF, although apparently correct, is somehow misused by Vault resulting in Vault being unable to connect to Kube API throwing “Unknown CA” errors.

When I then go to Vault web UI and change the CA there manually (“enter as text”) to the same value, everything is fixed immediately.

When I then re-run terraform apply, I can see there is a change in the CA and Terraform wants to update the value again. I say yes, Terraform deploys the change. I’d expect the connection from Vault to K8s break again but this time nothing changes, Vault still connects and there are no issues.

I tried running terraform apply before I changed the CA in Vault manually but at that point Terraform doesn’t see any changes - that’s what confusing me most: no matter how many times I re-run terraform apply, there are no changes until I “fix” the CA manually in Vault and any change “after” the fix doesn’t break Vault->K8s connection.

Relevant TF plan part:

data "kubernetes_config_map" "cacert" {
  metadata {
    name      = "kube-root-ca.crt"
    namespace = "kube-system"
  }
}

resource "vault_kubernetes_auth_backend_config" "kube" {
  backend            = vault_auth_backend.kube.path
  kubernetes_host    = var.kube_url
  kubernetes_ca_cert = data.kubernetes_config_map.cacert.data["ca.crt"]
  token_reviewer_jwt = local.jwt
  issuer             = "https://kubernetes.default.svc.cluster.local"
}

After this I can literally take the value from Vault, enter it again manually and stuff fixes:

$ vault read auth/kube-test/config
Key                       Value
---                       -----
disable_iss_validation    false
disable_local_ca_jwt      false
issuer                    https://kubernetes.default.svc.cluster.local
kubernetes_ca_cert        -----BEGIN CERTIFICATE-----
<cert data in BASE64
-----END CERTIFICATE-----
kubernetes_host           https://vm-test-01:6443
pem_keys                  []

I can copy-paste the kubernetes_ca_cert value directly to Vault Web UI under <vault_url>/ui/vault/settings/auth/configure/kube-test/configuration and once I hit “Save”, everything magically falls together and starts working.

When I then re-run terraform apply:

  # vault_kubernetes_auth_backend_config.kube will be updated in-place
  ~ resource "vault_kubernetes_auth_backend_config" "kube" {
        id                     = "auth/kube-test/config"
      ~ issuer                 = "" -> "https://kubernetes.default.svc.cluster.local"
      ~ kubernetes_ca_cert     = <<-EOT
            -----BEGIN CERTIFICATE-----
            <the very same base64 data as above >
            -----END CERTIFICATE-----
        EOT
        # (6 unchanged attributes hidden)
    }

Terraform and Vault versions:

$ tf --version
Terraform v1.1.9
on linux_amd64
+ provider registry.terraform.io/hashicorp/kubernetes v2.10.0
+ provider registry.terraform.io/hashicorp/vault v3.5.0

$ vault --version
Vault v1.9.2 (f4c6d873e2767c0d6853b5d9ffc77b0d297bfbdf)
2

so I tried to use a different source of the CA - Pulled the file from the remote Master node to TF directory and used that file as the source instead of the data resource. No change at all. Even though the CA data is correct, Vault doesn’t connect until I manually update the CA vaule in Vault. Consecutive terraform apply's don’t have any (negative) effect.

3

Never mind → RTFM! Kubernetes - Auth Methods | Vault by HashiCorp :slight_smile:
though it’s funny that disable_iss_validation was “automagically” set by Vault after intervention and then terraform respected that change and actually made it part of the plan (disable_iss_validation was never explicitly set in any .tf file)