Conditional kubernetes_manifest resource

Roxyrob · September 4, 2023, 11:23pm

Using resource:

resource “kubernetes_annotations” “example” {
count = var.create ? 1 : 0
provider = kubernetes.clus1
…
}

If var.create = false, k8s cluster will not be created and kubernetes_manifest resource return “…dial tcp [::1]:80: connect: connection refused”, so I need to comment out code using terraform kubernetes provider untile cluster is up.

It is possible to define “conditionally” kubernetes provider resource like kubernetes_manifest ?

maxb · September 5, 2023, 6:08am

No. I googled kubernetes_manifest and came across Avoid the Terraform kubernetes_manifest resource | by Daniel Jimenez Garcia | Medium which seems like a good description of the shortcomings and alternatives.

apparentlymart · September 5, 2023, 5:50pm

What you’ve described here seems like what the hashicorp/kubernetes provider tends to do when the host argument in its provider configuration has an unknown value: unfortunately, it treats it the same as the argument not being set at all, and tries to connect to localhost port 80 as a default.

This kubernetes_manifest resource type needs access to the cluster during planning because Kubernetes is a dynamic system which supports changing its schema at runtime, and so the provider needs to fetch the schemas from the live cluster in order to interpret and validate the manifest.

If that’s what’s going on here then I think you’ll need to split your Terraform configuration into two parts: one configuration which declares the kubernetes cluster (and its supporting infrastructure such as networks), and then another separate configuration which uses the hashicorp/kubernetes provider to declare objects that should exist in that cluster.

That kind of division also has the (perhaps marginal) advantage that you’ll presumably be making changes to the cluster and its infrastructure far less often than to objects inside the cluster, and so this separation will reduce the “blast radius” of changes to the second configuration.

If you expect to create your entire configuration “from scratch” only very rarely – that is, if the cluster is expected to run indefinitely once set up – another variation of the above would be to write everything in the same configuration but bootstrap with an extra step using -target to ask Terraform to work only on the cluster and its dependencies on the first run. For example, if you are using EKS (The AWS managed service) then you might bootstrap like this:

# First run focuses only on getting the cluster running.
# This will exclude everything that isn't needed to
# get the cluster created.
terraform apply -target=aws_eks_cluster.example

# Subsequent runs can just use Terraform normally as
# the cluster doesn't need to be recreated, since
# the cluster hostname will already be known and
# so the hashicorp/kubernetes provider will be able
# to configure itself for planning.
terraform apply

This situation is one of the examples included in the scope of this open design issue:

github.com/hashicorp/terraform

Unknown values should not block successful planning

opened 09:57PM - 26 Apr 22 UTC

apparentlymart

enhancement config unknown-values

The idea of "unknown values" is a crucial part of how Terraform implements plann…ing as a separate step from applying. An unknown value is a placeholder for a value that Terraform (most often, a Terraform provider) cannot know until the apply step. Unknown values allow Terraform to still keep track of type information where possible, even if the exact values aren't known, and allow Terraform to be explicit in its proposed plan output about which values it can predict and which values it cannot. Internally, Terraform performs checks to ensure that the final arguments for a resource instance at the apply step _conform to_ the arguments previously shown in the plan: known values must remain exactly equal, while unknown values must be replaced by known values matching the unknown value's type constraint. Through this mechanism, Terraform aims to promise that the apply phase will use the same settings as were used during planning, or Terraform will return an error explaining that it could not. > (For a longer and deeper overview of what unknown values represent and how Terraform treats them, see my blog post _[Unknown Values: The Secret to Terraform Plan](https://log.martinatkins.me/2021/06/14/terraform-plan-unknown-values/)_.) The design goal for unknown values is that Terraform should always be able to produce some sort of plan, even if parts of it are not yet known, and then it's up to the user to review the plan and decide either to accept the risk that the unknown values might not be what's expected, or to apply changes from a smaller part of the configuration (e.g. using `-target`) in order to learn more final values and thus produce a plan with fewer unknowns. However, Terraform currently falls short of that goal in a couple different situations: * The Terraform language runtime does not allow an unknown value to be assigned to either of the two resource repetition meta-arguments, [`count`](https://www.terraform.io/language/meta-arguments/count) and [`for_each`](https://www.terraform.io/language/meta-arguments/for_each). In that situation, Terraform cannot even predict how many instances of a resource are being declared, and it isn't clear how exactly Terraform should explain that degenenerate situation in a plan and so currently Terraform gives up and returns an error: ``` │ Error: Invalid for_each argument │ │ ... │ │ The "for_each" value depends on resource attributes that cannot │ be determined until apply, so Terraform cannot predict how many │ instances will be created. To work around this, use the -target │ argument to first apply only the resources that the for_each │ depends on. ``` ``` │ Error: Invalid count argument │ │ ... │ │ The "count" value depends on resource attributes that cannot be │ determined until apply, so Terraform cannot predict how many │ instances will be created. To work around this, use the -target │ argument to first apply only the resources that the count depends │ on. ``` * If any unknown values appear in a `provider` block for configuring a provider, Terraform will pass those unknown values to the provider's "Configure" function. Although Terraform Core handles this in an arguably-reasonable way, we've never defined how exactly a provider ought to react to crucial arguments being unknown, and so existing providers tend to fail or behave strangely in that situation. For example, some providers (due to quirks of the old Terraform SDK) end up treating an unknown value the same as an unset value, causing the provider to try to connect to somewhere weird like a port on localhost. Providers built using the modern Provider Framework don't run into that particular malfunction, but it still isn't really clear what a provider ought to do when a crucial argument is unknown and so e.g. the AWS Cloud Control provider -- a flagship use of the new framework -- reacts to unknown provider arguments by returning an error, causing a similar effect as we see for `count` and `for_each` above. Although the underlying causes for the errors in these two cases are different, they both lead to a similar problem: planning is blocked entirely by the resulting error and the user has to manually puzzle out how to either change the configuration to avoid the unknown values appearing in "the wrong places", or alternatively puzzle out what exactly to pass to `-target` to select a suitable subset of the configuration to cause the problematic values to be known in a subsequent untargeted plan. Terraform should ideally treat unknown values in these locations in a similar way as it does elsewhere: it should successfully produce a plan which describes what's certain and is explicit about what isn't known yet. The user can then review that plan and decide whether to proceed. Ideally in each situation where an unknown value appears there should be some clear feedback on what unknown value source it was originally derived from, so that in situations where the user doesn't feel comfortable proceeding without further information they can more easily determine how to use `-target` (or some other similar capabililty yet to be designed) to deal with only a subset of resources at first and thus create a more complete subsequent plan. --- This issue is intended as a statement of a problem to be solved and not as a particular proposed solution to that problem. However, there are some specific questions for us to consider on the path to designing a solution: * Is it acceptable for Terraform to produce a plan which can't even say how many instances of a particular resource will be created? That's a line we've been loathe to cross so far because the difference between a couple instances and tens of instances can be quite an expensive bill, but the same could be said for other values that Terraform is okay with leaving unknown in the plan output, such as the "desired count" of an EC2 autoscaling group. Maybe it's okay as long as Terraform is explicit about it in the plan output? A particularly "interesting" case to consider here is if some instances of a resource already exist and then subsequent changes to the configuration cause the `count` or `for_each` to become _retroactively_ unknown. In that case, the final result of `count` or `for_each` could mean that there should be _more_ instances of the resource (create), _fewer_ instances of the resource (destroy), or no change to the number of instances (no-op). I personally would feel uncomfortable applying a plan that can't say for certain whether it will destroy existing objects. * Conversely, is it acceptable for Terraform to _automatically_ produce a plan which explicitly covers only a subset of the configuration, leaving the user to run `terraform apply` again to pick up where it left off? This was essence of the earlier proposal #4149, which is now closed due to its age and decreasing relevance to modern Terraform. That proposal made the observation that, since we currently suggest folks work around unknown value errors by using `-target`, Terraform could effectively synthesize its own `-target` settings to carve out the maximum possible set of actions that can be taken without tripping over the two problematic situations above. * Should providers (probably with some help from the Plugin Framework) be permitted to return an _entirely-unknown_ response to the `UpgradeResourceState`, `ReadResource`, `ReadDataSource`, and `PlanResourceChange` operations for situations where the provider isn't configured completely enough to even _attempt_ these operations? These are the four operations that Terraform needs to be able to ask a partially-configured provider to perform. If we allow a provider to signal that it isn't configured enough to even try at those, what should Terraform Core do in order to proceed with that incomplete or stale information? * We most frequently encounter large numbers of unknown values when planning the initial creation of a configuration, when nothing at all exists yet. That is definitely the most common scenario where these problems arise, but a provider can potentially return unknown values even as part of an in-place update if that is the best representation of the remote API's behavior -- for example, perhaps one of the output attributes is derived from an updated argument in a way that the provider cannot predict or simulate. Do we need to take any extra care to deal with the situation where an unknown value cascades downstream from an _updated_ or _replaced_ resource instance? For example, if I've used an attribute from a vendor-specific Kubernetes cluster resource to provide an API URL to the `hashicorp/kubernetes` provider and the user changes the configuration of the cluster itself in a way that causes the API URL to change, how should Terraform and the Kubernetes provider react to the cluster URL being unknown even though there are existing objects bound to resources belonging to that provider which we will need to refresh and plan? * What sort of analysis would we need to implement in order to answer questions like "_why_ is this value unknown?" and "what subset of actions could I take in order to make this value be known?"?.

Dealing with the various situations where it isn’t even possible to plan until a value is unknown is an open area of research for me which I’m working on intermittently between other work. I don’t have any concrete details to share yet, but hopefully we’ll be able to improve on this eventually be adding some new concepts to Terraform’s execution model.

In the meantime, manually splitting configurations so that you can explicitly control the order that you plan and apply them is the best workaround we have.

Roxyrob · September 6, 2023, 8:33am

@apparentlymart, I think Terraform is fine to “provision k8s” clusters and base platform specific resources (e.g. AWS EKS + CNI, CSI, etc.) but GitOps solutions are the natural ways to “manage k8s” platform and application resources (controllers + convergence mechanism). So probably a minimal use of Terraform code is needed for edge cases and probably to create the Bridge between the two worlds (EKS Blueprint for terraform is an interesting sample/project).

Based on Terraform Kubernetes provider design/implementation limits, the separation of k8s cluster creation code and configuration code would be a solution but I faced this issue using the really good module “terraform-aws-modules/terraform-aws-eks”, some info in this related request in Terraform Kubernetes Provider project "Conditional kubernetes provider resources #2276 that clearly cannot be separated and make the module a little less flexible.

Probably a minimal use of solutions like “kubectl_manifest” only when really needed can be a solution for me.

Thank you for your comment and the really interesting links on “unknown values” discussions that definitely provided me useful answers to my question.