I’m trying to provision my environment using terraform, it creates everything from scratch, so in case of disaster I can simply run terraform apply and create all AWS and K8s resources I need.
Unfortunately, I keep hitting same wall. Some resources need to be created before others, furthermore some resources need to be created before planning changes in other resources. Terraform’s PLAN everything > APPLY everything model, does not work very well for me. What seems to be missing (or I don’t know how to set it up) is multiple plan+apply > plan+apply runs so I can create basic infrastructure, deploy next tier and then next tier.
For example, I have
aws_eks_cluster resource to provision EKS cluster where my application will run and I have
kubernetes_deployment resource for my application.
Logically, I can’t plan
aws_eks_cluster created, because terraform can’t connect to EKS cluster to check what exists and what not. It somehow works when nothing exists, because Kubernetes resources don’t exist on first run, then EKS cluster is provisioned and resources are created… but it quickly fails if terraform decides to re-create EKS cluster, because during planning terraform sees Kubernetes resources and thinks “ok, everything is up-to-date”, then during apply terraform kills cluster and does not re-create Kubernetes resources (they were ok during planning).
Another example, is
count resources, terraform can’t plan these without knowing
count values, which makes sense. You have to run terraform apply with
-target to create dependant resources first. It makes sense but shows same issue, you need to create some resources before planning others.
Another example, planning and creating
rabbitmq_user resources after provisioning
kubernetes_deployment for rabbitmq server. I hit same problem again and again.
I tried to use Terraform Cloud, but it simply does not work because there is no way to use
Then, i tried to run terraform with
-target, but it’s very complicated, you need to keep track of “first-tier resources” so you can provision them first (like eks cluster, or resources used in for_each or count operations).
I also tried to separate everything into independent terraform folders
terraform apply A,
terraform apply B but it created even more mess because there is no way to pass parameters between two terraform folders and I have my application tf code spread among multiple folders: create SSL certificate in one folder, create SSL verification DNS records in other.
Modules don’t work because within one run terraform does PLAN everything + APPLY everything.
Maybe there is a way to handle this, and I would really appreciate if somebody could point it out. But it seems to me terraform lacks some sort of staging system which would PLAN stage 1 > APPLY stage 1 > Plan Stage 2 > Apply Stage 2, everything within single terraform run so I can pass variables around and provision modules partially in each stage.
The way I imagine it with multiple stages (ideally running in Terraform Cloud) would be something like:
$ terraform apply Plan stage 1: VPC, EKS etc Apply stage 1: VPC, EKS etc Plan stage 2: k8s workloads Apply stage 2: k8s workloads
This way if on stage 1 EKS cluster is re-created, terraform will get correct state during stage 2 planning, so stage 2 apply will create missing resources. Maybe it can be achieved with some sort of