How do I configure a basic azurerm_kubernetes_cluster to use a basic-sku azurerm_public_ip and azurerm_lb?

Terraform Providers Azure
,
1

I’m having trouble configuring my aks cluster to use the load balancer and public ip address I created to use with it.

I have created an aks cluster with load_balancer_sku = basic as well as a basic-sku public IP address and basic-sku load balancer with the following code:

# azurerm_kubernetes_cluster
resource "azurerm_kubernetes_cluster" "k8s" {
  location            = azurerm_resource_group.rg.location
  name                = var.name
  resource_group_name = azurerm_resource_group.rg.name
  node_resource_group = "${local.node_resource_group_name}"
  dns_prefix          = "${local.dns_prefix}"

  identity {
    type = "SystemAssigned"
  }

  default_node_pool {
    name       = "agentpool"
    vm_size    = "Standard_B2s"
    node_count = 1
  }

  network_profile {
    network_plugin = "kubenet"
    load_balancer_sku = "basic" # ref: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-overview#pricing-and-sla
  }
}

# azurerm_public_ip
resource "azurerm_public_ip" "k8s_lb_ip" {
  name                = "PublicIPForLB"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  allocation_method   = "Static"
}

# azurerm_lb
resource "azurerm_lb" "k8s_lb" {
  name                = "BasicLoadBalancer"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name

  frontend_ip_configuration {
    name                 = "PublicIPAddress"
    public_ip_address_id = azurerm_public_ip.k8s_lb_ip.id
  }
}

The documentation states that I need to delegate that the cluster has the permissions to interact with the LB and Public IP, but isn’t clear on whether I can do this with the identity created via the identity { type = systemAssigned} or if I need to create a separate service principal; nor is it clear if I need ‘Contributor’ or ‘Network Contributor.’ So I use the following code, (which parallels letting the cluster interact with ACR):

# azurerm_role_assignment
# ref: https://learn.microsoft.com/en-us/azure/aks/static-ip?source=recommendations#create-a-service-using-the-static-ip-address
resource "azurerm_role_assignment" "lb_ip_aks_integration" {
  description = "Allows a basic-sku load balancer and basic-sku Public IP integration with an AKS cluster whose load_balancer_sku = basic."
  principal_id                     = azurerm_kubernetes_cluster.k8s.kubelet_identity[0].object_id
  role_definition_name             = "Network Contributor"
  scope                            = azurerm_public_ip.k8s_lb_ip.id #should the scope be the IP? the LB? the resource group where they are installed?
  skip_service_principal_aad_check = true

However, this does not work. A custom load balancer and custom public ip are indeed created in my resource group. But, in the auto-created and auto-managed “MC_” node resource group an additional load balancer named ‘kubernetes’ and a public ip named ‘kubernetes-’ is created.

When I deploy an application, yet ANOTHER public_ip is address (that’s three!) is automatically created, and then that ip is used. Take this code as an example. Notice I try to assign the public ip of my custom load balancer explicitly, but that makes no difference and it’s ignored.

resource "helm_release" "nginx_ingress" {
  name = "nginx-ingress-controller"

  repository = "https://charts.bitnami.com/bitnami"
  chart      = "nginx-ingress-controller"

  set {
    name  = "controller.service.externalTrafficPolicy"
    value = "Local"
  }
  set {
      name        = "controller.service.loadBalancerIP"
      value = azurerm_public_ip.k8s_lb_ip.ip_address
   }
}

What am I doing wrong? How do I create a custom Public IP address and Load balancer and ensure that the cluster created with azurerm_kubernetes_cluster uses only those resources and, preferably, does not keep creating other resources automatically?

1 Like