I have seen this issue brushed aside for years, and I genuinely don’t understand why. I opened this feature request, but it was closed. My query is simple.
Imagine you want a module to create a policy resource only if a policy is passed in. This can be done simply with:
This works well and is simple to understand. However, if that policy document contains any data that cannot be known until apply-time, it fails The "count" value depends on resource attributes that cannot be determined until apply. That makes sense in some cases, but not in this case. The value is not null. So, the count should be 1. I really struggle to understand why this simple statement cannot be used.
This works just fine. But in both cases, the obvious fact is that they are not null. We can see this as humans? It isn’t relevant to the count attribute that the value isn’t known yet; it should still be 1.
Question 1: What is the best practice workaround?
Question 2: Why can this not be implemented and supported by terraform?
It looks like in subsequent discussion on that issue you already saw this other issue where I left a comment describing why this occurs and what we might need to add to the language in order to support what you tried:
In your case there is a chain of events which leads to this problem indirectly:
some_resource.example.arn won’t be known until the apply step, because the provider marked it as “unknown” in the plan.
The aws_iam_policy_document needs to pack everything into a single big string containing a JSON IAM policy document. Terraform doesn’t have any sense of a string that has some known parts and some unknown parts, so that entire result must therefore be unknown as long as any part of it would be unknown.
Although we intuitively know that data.aws_iam_policy_document.this.rendered will always turn out to be a non-null string, there’s nothing in Terraform’s type system which guarantees that – the provider could potentially finally decide to set that attribute to null after the apply step, and indeed there are real examples in other resource types of attributes where that is true. Therefore data.aws_iam_policy_document.this.rendered != null (which is what you effectively did there, even though you had an input variable in the middle) is also unknown; Terraform doesn’t have enough information to prove either way whether this value will be null.
In order for this to work as expected then, we would need to have a way for the provider schema for aws_iam_policy_document to state that rendered is guaranteed to never be null, and for Terraform to then somehow carry that information through other expressions so that var.policy != null can eventually know that var.policy is derived from something which was guaranteed not to be null.
For now I’ve typically been recommending wrapping the unknown value in a container which will itself be null in the unset case, which then separates the information about what the value is from the information about whether the argument is set. In your case that would involve redefining this input variable to be some sort of container type, such as an object type:
In the above example, the null-ness of var.policy is what decides the condition, regardless of what value var.policy.json has. From a caller’s perspective if they write out the object wrapper literally then it will be guaranteed always known and never null and so this would achieve the desired result:
Even if var.policy.json is an unknown value, we can still see that var.policy is a non-null object value.
The approach you showed here using for_each is interesting but I feel a little unsure about whether it’s working the way you think it is. You still have a var.policy != null expression in there, which will still return an unknown value in this scenario, and therefore your for expression is effectively { for i, v in <unknown value> : ... } (pseudocode) and so therefore its own result would be an unknown value and thus invalid for for_each. Are you sure that when you saw that work you hadn’t tested it in a situation where the ARN was already known from a previous run? (If that was true then your simpler var.policy_doc != null ? 1 : 0 would’ve worked in that situation too.)
First, thank you for taking the time to explain this thoroughly. I hadn’t yet seen #26755 when I opened this. That and your explanation here help clarify why it is the way it is. My position is simply that our intuition about the logic here seems to warrant a solution that behaves as a user would expect; some mechanism to guarantee a value will be not null, as you say.
You solution of housing the value in an object type makes sense, but it does, I think at least a little, diminish the clarity of the input variable; or at least adds seemingly unnecessary bits to it. It does make sense though.
Regarding my for_each cheat, I really do think it works though I expected it to fail for the reason you mention. I tested it with and without the input variable and it behaved. I will see if I can implement another minimal solution to prove or disprove its effectiveness.
I have written a better test to verify what sure felt like it was working earlier. As it turns out, you were dead on, as I should have expected. The for_each cheat does not work, at least in some cases. Perhaps the arn was known already in this scenario.
Nevertheless, the object wrapper does work, and it works very well. It feels slightly less than ideal, but I’m over it. A thousand thanks.
FWIW, I very much agree that I would like this to work better; it’s only that I don’t know how best to get there from here, since a type system with explicit nullability being part of the type is quite a different beast than our current type system where null just just a value that can be converted to any type.
The design of Terraform’s null here essentially follows how it behaves in various mainstream languages we expected Terraform users might have prior familiarity with, such as Python (with None) and JavaScript. Unfortunately Terraform has this additional idea of values being unknown that neither of those languages have.
The idea of unknown values does encourage encoding more information into the types than other languages might, so that we can retain that information even when a value isn’t known, but the last time we significantly grew the scope of our type system was in Terraform v0.12 and that led to a significant ecosystem break that we cannot repeat within the constraints of the Terraform v1.0 Compatibility Promises.
It’s unfortunate that we also added null in Terraform v0.12 and so we didn’t get the opportunity to get this experience with the interaction between null and unknown values before we made the improvements to the type system, or else we probably would’ve seen the benefits of explicitly-nullable types and decided that the benefits would outweigh the cost of a type system with features less likely to be familiar to new users. Prior to v0.12 people had already been using zero-or-one-element lists as a substitute for null, which functionally equivalent to the possibly-null object solution I shared previously but arguably even less ergonomic, and so here I essentially just “modernized” that old approach making use of our new concept of object types, instead of using lists.
Hopefully we’ll find a reasonable compromise for this eventually, but I don’t yet have one and so although I can see an ideal state we’d like to reach, we currently lack a viable path to get there.
Could we maybe get an isnull() function, similar to various databases, which returns true or false based on if the parameter is null or not? Presumably that could return a value regardless of if any of the contained values are not known (because it only is checking the top part is null or not, so an object of any type [containing null keys or unknown ones] would immediately return false as it isn’t null)
As far as I can tell, isnull(...) would not be any different than ... == null, which already checks only if the top-level value is null.
unknown == null returns unknown because of the top-level unknown. [unknown] == null (that is, a tuple or list with an element that is unknown) returns false because the list itself is known to not be null. This is true for any other type which acts as a container for others.
(Terraform 0.11 and earlier did have the limitation that a collection containing at least one unknown value would reduce to being wholly unknown itself, but that was one of the things we addressed in the rewrite for 0.12.)
If you used isnull(unknown) = false logic to generate a plan, then that result would change during apply once the value is known. This can cause various types of errors depending on where it was used, but in general it would cause the apply to fail.
I guess there is the rare occasion that the unknown could result in null (although I’d expect most of those possibilities, such as a datasource searching for and failing to find something, would probably be resolved during the plan)
It’s indeed unlikely that an id or arn attribute would become null after being unknown during planning, but there are various other examples of attributes unknown during planning which can plausibly become null during the apply step.
For example, a provider which is representing virtual machines may not know until the apply step whether the VM would be assigned an IPv6 address, if the assignment depends on the configuration of the network the VM is being launched into, and so this hypothetical provider would return a hypothetical ipv6_address attribute which would be unknown during planning and then either a known string or null at apply time, depending on that result.
It’s providers which make the decisions here, which is the reason for my earlier assertion that this would need to be something providers can specify either in their schema (statically) or in the plan response (dynamically), which Terraform Core would then somehow track through references so that operations like <unknown> == null can return false if the unknown value is provably non-null.
An important rule that Terraform’s planning behavior relies on is that any expression that produces a known result during planning must always produce the same result during apply, even when the apply step makes more information available. Without this, the behavior during apply can be significantly different than what was planned. It would therefore only be safe for <unknown> == null or (equivalently) isnull(<unknown>) to return a known value if we can guarantee that it would return the same known value once that unknown value becomes a known final value.
As it turns out, you were dead on, as I should have expected. The