For_each hack claims key unknown at plan-time; seems incorrect

Terraform
1

I’m trying to understand why one of these solutions works and the other doesn’t. In counts like these:

  count = var.x != null ? 1 : 0
  # or
  count = var.y != null && var.z != null ? 1 : 0

there often arises the age-old unknown at plan-time issue. To get around this, one can use for_each and ensure the key is not a calculated value.

For example, in the former case:

  for_each = { for i, v in [var.x] : i => v }

works nicely, and no longer complains. You just get key ["0"] instead of [0].

But these don’t work for the second count.

  for_each = { for i, v in [{
    y = var.y
    z = var.z
    }] : i => v if alltrue([
    v.y != null,
    v.z != null
  ]) }

and:

  for_each = { for i, v in [alltrue([
    var.y != null,
    var.z != null
    ])] : i => {
    y = var.y
    z = var.z
  } if v }

They yield the error:

The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of
│ keys that will identify the instances of this resource.

But this works:

  for_each = { for i, v in [alltrue([
    var.y != null,
    var.z != null
    ]) ? {
    y = var.y
    z = var.z
  } : {}] : i => v }

But given that the keys are the same and the known information is the same, I cannot see why the first two raise the error but the third one doesn’t. Not pressing as all are ugly hacks and one is working, but I am confused by the error.

2

I was actually wrong. Non of these work as expected in all cases. Terraform will tell you the keys are based on information that cannot not be known even though the length of a list can very be know. It is, in my view, the most frustrating shortcoming in terraform after years of use.

3

If you provide a simple single reproduction case that demonstrates the problem you are experiencing, it is more likely someone will be able and willing to engage with your post.

Currently you’ve presented so many options, and are not entirely clear about which ones work and don’t work given what inputs, that this seems too uncertain for me to want to get involved.

4

Sure, I should be more clear. Also, I didn’t mean to sound grumpy or cross; I just regularly get bitten by this issue. Consider a root module:

resource "aws_sns_topic" "test" {
  name = "h4s-test"
}

# Yields apply-time error. This sort of makes sense, but is still a bummer.
module "just_count" {
  source = "./just-count"

  topic_arn = aws_sns_topic.test.arn
}

# Works as expected.
module "for_each_list" {
  source = "./for-each-list"

  topic_arn = aws_sns_topic.test.arn
}

# Creates, but shouldn't since the value is null.
# for_each = { for i, v in [var.topic_arn] : i => v if v != null }
# would again add the apply-time error, even though the map key is
# not a calculated value.
module "for_each_list_but_null" {
  source = "./for-each-list"
}

# This one is the most confusing. Keys not known until apply? How?
# The topic definitely will be created, so there will be an arn. The contents
# of the arn are not a key value here. They key value will either be 0 or it
# will not exist, and this is known at plan time.
module "for_each_list_alternate" {
  source = "./for-each-list-alternate"

  topic_arn = aws_sns_topic.test.arn
}


# Works as expected, but requires less simple input.
module "count_with_object" {
  source = "./count-with-object"

  topic = { arn = aws_sns_topic.test.arn }
}

Then the following modules:

./just-count/main.tf

variable "topic_arn" {
  default = null
  type    = string
}

resource "null_resource" "this" {
  count = var.topic_arn != null ? 1 : 0
}

./for-each-list/main.tf

variable "topic_arn" {
  default = null
  type    = string
}

resource "null_resource" "this" {
  for_each = { for i, v in [var.topic_arn] : i => v }
}

./for-each-list-alternate/main.tf

variable "topic_arn" {
  default = null
  type    = string
}

resource "null_resource" "this" {
  for_each = { for i, v in var.topic_arn != null ? [var.topic_arn] : [] : i => v }
}

./count-with-object/main.tf

variable "topic" {
  default = null
  type    = object({ arn = string })
}

resource "null_resource" "this" {
  count = var.topic != null ? 1 : 0
}

This last one seems to be the best workaround, and I have discussed it in the past with @apparentlymart. The issue in this topic, is that I thought I had come up with a workaround that in some cases allowed the module caller to use the more simple variable and still not face the plan-time issue, but there was a strange behavior that didn’t make sense.

However, it turns out that the workarounds didn’t actually work in all cases, so I just wanted to update that I was wrong, so people didn’t find this and expect them to work.

I hope these examples clarify the issue, so any discussion on it can be more fruitful.

5

The most simple summary of the intent is: Create a resource given a nullable value variable only if that value is given, without regard for the contents of that value, avoiding the “unknown until apply” issue.

6

I think what you are noticing is that if a value is unknown then Terraform doesn’t know whether it is null or not, because null-ness in Terraform is tracked only for specific values, and not as part of the type system.

In essence: Terraform treats all types as “a value of this type or null” and has no sense of something being guaranteed non-null as part of its type. Therefore when we have an unknown value (which means we only know a type constraint, and not an actual value) there is no way to prove whether it could be null or not once the value is finally decided.

It’s frustrating because we know from the provider behavior that an ARN can never be null in practice, but Terraform can only see the type information and not the implementation behavior and so it needs to be conservative. Improving this would require retrofitting “nullable types” into the type system, but it isn’t yet clear how to do that while remaining backwards compatible.