Kubernetes resources destroyed too late (after worker_group & fargate)

Terraform HCP Terraform
1

Hi,

We’re creating aws environment through terraform. So in tf we first setup VPC then EKS cluster (with worker_group & fargate) and then kubernetes objects.

We did split everything into Modules so VPC & EKS and kubernetes are each separated

We did setup a dependency between eks & kubernetes module by using a trigger on a null resource which is triggered by an output variable of the eks module “wait_for_eks” like this:

resource "null_resource" "dependency" {
  triggers = {
    dependency_id = var.wait_for_eks
  }
}

resource "kubernetes_namespace" "my_namespace" {
  metadata {
    name = "app_namespace"
  }
  depends_on = [null_resource.dependency]
}

The variable wait_for_eks is passed into the kubernetes module as an output from eks module which just sets up a dummy string “finished” like this:

output "eks_done" {
  value       = "finished"
  description = "variable to trigger dependent kubernetes module"
}

and then passed along in the main.tf

module "kubernetes" {
  source                               = "./modules/kubernetes"
  wait_for_eks                         = module.eks.eks_done
  ...
}

This solved the problem of kubernetes provider creating objects too early before worker_group & fargate nodes are available. But on destruction terraform destroys the eks resources first. So it first deletes the managed & fargate nodes before it tries to delete the kubernetes resources and this of course then fails when the nodes were deleted already.

Actually I found this technique on making an indirect dependencies across modules in a terraform article where they also introduced the “depends_on” for modules which doesn’t work though if you have providers in that module that rely on the output of other modules (like in our case output data returned by the eks module which is needed by the kubernetes provider).

How can I solve this issue so that terraform first deletes the k8s objects before it tries to delete the worker nodes? Shouldn’t the deletion just work the opposite way so kubernetes_module which relies on eks module should be deleted first?

Thanks for any help on this because we are really stuck here and we destroy the env regularly but this now always requires manually tweaking and modifying state and remove kubernetes stuff.

2

We instead split into different state files/code repos, one which sets up the VPC & EKS and a totally separate one which installs things within the cluster.