For_each in resource that depends on a data item that depends on a resource

[dsargent3220]

I’ve seen some questions about this elsewhere but they do not seem to answer my issue (or I don’t understand the answer?)

Here’s the complete error.

│ Error: Invalid for_each argument
│
│   on longhorn_manifests.tf line 34, in resource "kubectl_manifest" "longhorn_manifests":
│   34:   for_each  = { for k, v in data.kubectl_path_documents.longhorn_directory.manifests : k => v if var.install_longhorn }
│     ├────────────────
│     │ data.kubectl_path_documents.longhorn_directory.manifests is a map of string, known only after apply
│     │ var.install_longhorn is true
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will
│ identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.

and here is the code

resource "random_password" "salt" {
  length = 8
}

resource "htpasswd_password" "hash" {
  password = var.longhorn_ui_password
  salt     = random_password.salt.result
}

data "kubectl_path_documents" "longhorn_directory" {
  pattern = "./manifests/longhorn/*.yaml"
  vars = {
    cluster_fqdn           = var.cluster_fqdn
    longhorn_s3_endpoint   = base64encode(var.longhorn_s3_endpoint)
    longhorn_s3_access_key = base64encode(var.longhorn_s3_access_key)
    longhorn_s3_secret_key = base64encode(var.longhorn_s3_secret_key)
    auth                   = base64encode("${var.longhorn_ui_user}:${htpasswd_password.hash.apr1}")
  }
}

resource "kubectl_manifest" "longhorn_manifests" {
  for_each  = { for k, v in data.kubectl_path_documents.longhorn_directory.manifests : k => v if var.install_longhorn }
  yaml_body = each.value

  depends_on = [
    kubectl_manifest.infoblox_issuers_manifests,
    kubernetes_namespace.longhorn,
    helm_release.longhorn
  ]
}

I think I understand the issue, I just don’t know how to fix it.

The auth = base64encode("${var.longhorn_ui_user}:${htpasswd_password.hash.apr1}") line is the problem, specifically the {htpasswd_password.hash.apr1} element.

I need to create the password hash using a resource, then use that in the vars section of the data item, then use that data item in a for_each of the kubectl_manifest.longhorn_manifests resource. From the error it sounds like the data item needs the resource to run first and it can’t

I get the second option for a fix, perform two applies, but that seems really hacky and not ideal.
I don’t get the first option, it's better to define the map keys statically. I’ve searched for examples but have come up with nothing.

Notes:
htpasswd_password is from loafoe/htpasswd
kubectl_path_documents is from alekc/kubectl

Any ideas?

[apparentlymart]

Hi @dsargent3220,

When the error message refers to “define the map keys statically” it means that the set of keys would be defined as a value within the configuration itself, rather than relying on an external system.

That’s tricky to apply in this case because the kubectl_path_documents data source is the one deciding what the map keys of its manifest attribute are. Therefore the only way to define this “statically” would be to partially implement some of the work that the data source is doing as code in your own module that refers only to locally-available data.

However, after spending a little time reviewing the implementation of kubectl_path_documents it seems like it’s at least mostly implementable using features built right into Terraform: find files matching a glob pattern, evaluate them as templates with a specified set of variables, and then gather them into a map.

The parts that are not so easy to achieve in a Terraform module are:

1. Splitting a multi-document YAML blob into its separate documents.
2. Some sort of intermediate manifest decoding and re-encoding that I didn’t delve into the details of; I guess this is probably doing some static validation of the manifest structure?
3. Constructing an idiomatic Kubernetes-style “self link” for the object that each manifest is describing, as implemented in buildSelfLink.

The kubectl_manifest resource type seems to repeat all of these except the last step itself anyway though, so it seems like the main loss in reimplementing this data source “manually” in your module would be that it would be hard to make the for_each keys appear as “self links”. But if you don’t care what those instance keys are and just want these things to get declared somehow then you could potentially use the YAML filename as the unique identifier, at which point I think you could do something like the following:

locals {
  manifests = {
    for fn in fileset("${path.module}/manifests/longhorn", "*.yaml") :
    fn => templatefile("${path.module}/manifests/longhorn/${fn}", {
      cluster_fqdn           = var.cluster_fqdn
      longhorn_s3_endpoint   = base64encode(var.longhorn_s3_endpoint)
      longhorn_s3_access_key = base64encode(var.longhorn_s3_access_key)
      longhorn_s3_secret_key = base64encode(var.longhorn_s3_secret_key)
      auth                   = base64encode("${var.longhorn_ui_user}:${htpasswd_password.hash.apr1}")
    })
  }
}

resource "kubectl_manifest" "longhorn_manifests" {
  for_each = {
    for k, v in local.manifests : k => v
    if var.install_longhorn
  }

  yaml_body = each.value

  depends_on = [
    kubectl_manifest.infoblox_issuers_manifests,
    kubernetes_namespace.longhorn,
    helm_release.longhorn
  ]
}

With this structure, the keys in local.manifests are decided by Terraform itself based on the files it finds in manifests/longhorn, and so those keys should all be known during the planning phase. The values in that map are likely to be unknown strings during an initial create, but the only potential problem that causes is that Terraform won’t be able to tell you what value is being assigned to the yaml_body argument; it should still allow you to complete planning and decide for yourself if you are comfortable with that uncertainty.

The local.manifests values are not guaranteed to contain valid YAML syntax nor valid structure for a Kubernetes manifest, and so you’ll be relying on the kubectl_manifest resource type to validate those during the apply phase.

The instance keys of kubectl_manifest.longhorn_manifests will be named after the file the manifest was loaded from, rather than after the remote Kubernetes object that the manifest describes. In particular, this means you won’t get any local validation that there aren’t two manifests trying to declare the same remote object.

[dsargent3220]

That did it!
Thank you so much, I would have never figured that out on my own!