If var.create = false, k8s cluster will not be created and kubernetes_manifest resource return “…dial tcp [::1]:80: connect: connection refused”, so I need to comment out code using terraform kubernetes provider untile cluster is up.
It is possible to define “conditionally” kubernetes provider resource like kubernetes_manifest ?
What you’ve described here seems like what the hashicorp/kubernetes provider tends to do when the host argument in its provider configuration has an unknown value: unfortunately, it treats it the same as the argument not being set at all, and tries to connect to localhost port 80 as a default.
This kubernetes_manifest resource type needs access to the cluster during planning because Kubernetes is a dynamic system which supports changing its schema at runtime, and so the provider needs to fetch the schemas from the live cluster in order to interpret and validate the manifest.
If that’s what’s going on here then I think you’ll need to split your Terraform configuration into two parts: one configuration which declares the kubernetes cluster (and its supporting infrastructure such as networks), and then another separate configuration which uses the hashicorp/kubernetes provider to declare objects that should exist in that cluster.
That kind of division also has the (perhaps marginal) advantage that you’ll presumably be making changes to the cluster and its infrastructure far less often than to objects inside the cluster, and so this separation will reduce the “blast radius” of changes to the second configuration.
If you expect to create your entire configuration “from scratch” only very rarely – that is, if the cluster is expected to run indefinitely once set up – another variation of the above would be to write everything in the same configuration but bootstrap with an extra step using -target to ask Terraform to work only on the cluster and its dependencies on the first run. For example, if you are using EKS (The AWS managed service) then you might bootstrap like this:
# First run focuses only on getting the cluster running.
# This will exclude everything that isn't needed to
# get the cluster created.
terraform apply -target=aws_eks_cluster.example
# Subsequent runs can just use Terraform normally as
# the cluster doesn't need to be recreated, since
# the cluster hostname will already be known and
# so the hashicorp/kubernetes provider will be able
# to configure itself for planning.
This situation is one of the examples included in the scope of this open design issue:
Dealing with the various situations where it isn’t even possible to plan until a value is unknown is an open area of research for me which I’m working on intermittently between other work. I don’t have any concrete details to share yet, but hopefully we’ll be able to improve on this eventually be adding some new concepts to Terraform’s execution model.
In the meantime, manually splitting configurations so that you can explicitly control the order that you plan and apply them is the best workaround we have.
@apparentlymart, I think Terraform is fine to “provision k8s” clusters and base platform specific resources (e.g. AWS EKS + CNI, CSI, etc.) but GitOps solutions are the natural ways to “manage k8s” platform and application resources (controllers + convergence mechanism). So probably a minimal use of Terraform code is needed for edge cases and probably to create the Bridge between the two worlds (EKS Blueprint for terraform is an interesting sample/project).
Based on Terraform Kubernetes provider design/implementation limits, the separation of k8s cluster creation code and configuration code would be a solution but I faced this issue using the really good module “terraform-aws-modules/terraform-aws-eks”, some info in this related request in Terraform Kubernetes Provider project "Conditional kubernetes provider resources #2276 that clearly cannot be separated and make the module a little less flexible.
Probably a minimal use of solutions like “kubectl_manifest” only when really needed can be a solution for me.
Thank you for your comment and the really interesting links on “unknown values” discussions that definitely provided me useful answers to my question.