Kubernetes cluster unreachable when I changed the vm_size in azurerm_kubernetes_cluster

Terraform Providers Azure
,
1

Version

terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "=3.16.0"
    }
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "=2.11.0"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "=2.6.0"
    }
  }

  required_version = "=1.2.6"
}

Code

resource "azurerm_kubernetes_cluster" "my_cluster" {
  name                      = local.cluster_name
  location                  = azurerm_resource_group.rg.location
  resource_group_name       = azurerm_resource_group.rg.name
  dns_prefix                = local.dns_prefix
  node_resource_group       = local.resource_group_node_name
  kubernetes_version        = "1.24.3"
  automatic_channel_upgrade = "patch"
  sku_tier                  = var.sku_tier

  default_node_pool {
    name    = "default"
    type    = "VirtualMachineScaleSets"
    vm_size = var.default_pool_vm_size

    enable_auto_scaling = true
    max_count           = var.default_pool_max_count
    min_count           = var.default_pool_min_count

    os_disk_type    = "Ephemeral"
    os_disk_size_gb = var.default_pool_os_disk_size_gb
  }

  identity {
    type = "SystemAssigned"
  }

  network_profile {
    network_plugin = "kubenet"
  }
}

provider "helm" {
  kubernetes {
    host                   = azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.host
    client_certificate     = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.client_certificate)
    client_key             = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.client_key)
    cluster_ca_certificate = base64decode(azurerm_kubernetes_cluster.my_cluster.kube_admin_config.0.cluster_ca_certificate)
  }
}

resource "helm_release" "argocd" {
  name             = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "4.10.5"
  create_namespace = true
  namespace        = "argocd"
}

Steps to Reproduce

  1. All resources were created successfully when I executed the terraform code at first time creation.
  2. But it was failed on terraform plan when I changed the vm_size of default node pool.

Error

$ terraform plan

Error: Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
with helm_release.argocd,
│   on argocd.tf line 1, in resource "helm_release" "argocd":
│    1: resource "helm_release" "argocd" {

Expected Behavior

The cluster should be reachable even the vm_size was changed.

Actual Behavior

Kubernetes cluster is unreachable for other provider (ex: kubernetes, helm)

Test

  1. I removed resource argocd to prevent the above situation, then terraform could plan and apply successfully.
  2. I get the cluster config data from azure portal, the azurePortalFQDN is different between first time creation.

Question

  1. The whole cluster will be recreated if I change default node pool config which is commented “Changing this forces a new resource to be created” on terraform documents.
  2. Why provider helm can connect to cluster at first time creation (the dependencies included azurerm_resource_group.rg and azurerm_kubernetes_cluster.my_cluster), but it failed when the resource recreation?

Thanks for your reply.

2 Likes