"Depends on" feature is not working properly to run the k8s manifests after EKS cluster creation

Terraform Providers Kubernetes
1

Hi folks!

I have been trying to run this terraform code to create the alb_controller and service account for it after the EKS cluster creation:

provider "kubernetes" {
  config_path    = "~/.kube/config"
  config_context = "${module.eks.cluster_arn}"
}

resource "kubernetes_manifest" "alb_controller" {
  manifest = {
    "apiVersion" = "v1"
    "kind"       = "ServiceAccount"
    "metadata" = {
      "name"      = "aws-load-balancer-controller"
      "namespace" = "kube-system"
      "labels"    = {
        "app.kubernetes.io/component" = "controller"
        "app.kubernetes.io/name" = "aws-load-balancer-controller"
      }
      "annotations" = {
        "eks.amazonaws.com/role-arn" = "${aws_iam_role.aws_alb_controller_iam_role.arn}"
      }
    }
  }
  depends_on = [module.eks.cluster_endpoint]
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    host                   = module.eks.cluster_endpoint
    cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
  }
}

resource "helm_release" "aws_alb_controller" {
  namespace        = "kube-system"
  create_namespace = false

  name       = "aws-load-balancer-controller"
  repository = "https://aws.github.io/eks-charts"
  chart      = "aws-load-balancer-controller"
  version    = "1.6.2"

  set {
    name  = "serviceAccount.name"
    value = "aws-load-balancer-controller"
  }
  set {
    name  =  "serviceAccount.create"
    value =  "false"
  }
  set {
    name  = "clusterName"
    value = module.eks.cluster_name
  }
  set {
    name  = "clusterEndpoint"
    value = module.eks.cluster_endpoint
  }

  depends_on = [kubernetes_manifest.alb_controller, aws_iam_policy.aws_alb_controller_iam_policy, module.eks.cluster_endpoint ]
}

But the terraform plan has been displaying this error message informing that cannot call the EKS API:

Plan: 81 to add, 0 to change, 0 to destroy.
╷
│ Error: Invalid configuration for API client
│
│   with kubernetes_manifest.alb_controller,
│   on aws-alb.tf line 66, in resource "kubernetes_manifest" "alb_controller":
│   66: resource "kubernetes_manifest" "alb_controller" {
│
│ Get "https://F2BDA1C615D3116C4D7BC8XXFC611134.gr7.us-east-1.eks.amazonaws.com/apis": dial tcp: lookup F2BDA1C615D3116C4D7BC8A0FC611134.gr7.us-east-1.eks.amazonaws.com on 172.XX.176.1:53: no such host

Could somebody help me to understand why it doesn’t work?

Thanks!

2

Hi @cvieiradas,

This situation arises because the kubernetes_manifest resource type is designed to use the remote API to create its plan. This is different to most other resource types, where planning is implemented entirely locally inside the provider based on a copy of the schema. Because Kubernetes is a dynamic system that allows reconfiguring schema at runtime, the provider must therefore fetch the schema from the live cluster in order to understand whether your manifest is valid.

Usually the Kubernetes cluster in a Terraform configuration gets recreated very infrequently relative to the other resources that use it, and so a possible way to proceed is to perform your initial creation in two steps:

terraform apply -target=module.eks
terraform apply

The first command asks Terraform to plan and apply only the resources in the “eks” module, and so it will not try to plan kubernetes_manifest.alb_controller. The second command then performs a normal full plan, allowing Terraform to complete all of the remaining actions once the EKS cluster is already running.