The behaviour of this new feature is exactly what I need. I have started targeting 1.3 for all my development work.
Thank you for this comprehensive answer.
Hi,
I’ve been experimenting with the previous defaults() function and came across the issue below so after reading through I have abandoned my experiment
This new functionality seems to solve most of the complaints in the GitHub issue.
One scenario I have that does not work with the “new” approach is like below where I have dynamic optional values
scaling : object({
enabled : optional(bool, local.scaling_enabled)
min_replicas : optional(number, 1)
max_replicas : optional(number, 5)
})
This example is simplified - my real type has multiple levels of optional nesting and a bit of conditional logic to set local.scaling_enabled.
If the defaults() function had worked with nested objects it would have been a good solution for this scenario.
Are there any plans to support var or local in the optional attribute?
Hi @danoliver1,
Unfortunately I don’t think it will be feasible to support dynamic defaults because Terraform tends to need to resolve default values early on during validation and type conversion, rather than at runtime in the actual plan and apply operations.
I think that would be a situation where you would still need a function like defaults. Although that particular function didn’t survive into the final release, the design of that function was essentially an extension of the existing coalesce function to try to apply it to entire data structures at once, and so you could potentially use coalesce for enabled specifically in order to get an equivalent result:
coalesce(var.something.scaling.enabled, local.scaling_enabled)
If enabled is null then coalesce will return local.scaling_enabled instead.
I expect we’ll try to improve on this situation in later releases, but I can’t say yet exactly what the design will be. We know from feedback that the defaults function was very confusing to use, so I don’t expect we’ll try to bring that back again, but we do still need to find some way that involves normal expressions evaluated at runtime, rather than static values embedded inside the type constraints.
Fortunately if we do find a suitable design later then I expect it will end up being backward-compatible with what we’ve stablized for v1.3, and so answering that question doesn’t need to be a blocker for concluding the experiment.
Is there a nullable equivalent that works for object attributes?
Hi @lorengordon,
There is not a mechanism exactly equivalent to “nullable”, because nullable was a compromise we were forced to take in order to remain backward-compatible with modules written before it existed.
Specifically, there is one case that has no equivalent in optional attributes: distinguishing between a caller explicitly assigning null and a caller just omitting an optional argument entirely. We consider it a historical design error that it was possible to distinguish those cases; assigning a null value to an optional input variable that specified a default should’ve caused the final value to be that default. nullable = false effectively opts in to that correct behavior, and disables the incorrectly-designed behavior.
For optional attributes, there is intentionally no way to distinguish those situations, because there is no regrettable existing behavior to be backward compatible with in this case; the caller explicitly assigning null is exactly equivalent to omitting the attribute altogether, which matches how arguments in resource blocks work. If the optional attribute has no default value then in both cases the attribute’s value will be null, and if there is a default then in both cases the attribute’s value will be the specified default value.
To say that in more concrete terms, the optional attribtues syntax allows you to specify behavior equivalent to the following two situations:
# This one is equivalent to an optional
# attribute with a default value:
# a = optional(string, "foo")
variable "a" {
type = string
default = "foo"
nullable = false
}
# This one is equivalent to an optional
# attribute without a default value:
# b = optional(string)
variable "b" {
type = string
default = null
# nullable = true is the default
}
# There is no equivalent to the following
# for an optional attribute, because this
# will produce "foo" if c is omitted, and
# null if the caller writes c = null .
variable "c" {
type = string
default = "foo"
# nullable = true is the default
}
# And for completeness, this last combination
# isn't valid for either variables or optional
# attributes:
variable "d" {
type = string
default = null
nullable = false
# INVALID because default = null contradicts
# nullable = false .
}
Thanks @apparentlymart! I don’t think I had seen that spelled out explicitly, so I was operating under the assumption that it worked just like variables. In other words, that the user could set null explicitly, and that null would then be the resulting value, even if the variable had a default value.
Naturally, I’m probably one of those that will run into this as an edge case, as I seem to have a perverse knack for inadvertently finding these seams and taking advantage of them! 
Martin’s post above is not completely correct: assigning null is actually treated differently from omitting the attribute altogether.
Your assumption was correct: this is how the feature works at present. Consider this configuration:
variable "foo" {
type = object({
a = optional(string, "hello"),
})
}
output "foo" {
value = var.foo
}
Omitting the a attribute here causes Terraform to apply the specified default value:
$ terraform plan -var 'foo={}'
Changes to Outputs:
+ foo = {
+ a = "hello"
}
Whereas supplying an explicit null value causes Terraform to pass it through:
$ terraform plan -var 'foo={ a = null }'
Changes to Outputs:
+ foo = {
+ a = null
}
This behaviour is intuitive to me, but clearly not so for everyone. I’d be interested to hear your thoughts, @lorengordon, as well as any feedback others may have.
Indeed, sorry for the misinformation above, everyone! This is a danger of extrapolating from details of earlier phases of the design without checking my work first.
You can see from this that we’re also not entirely sure what the best tradeoff is between flexibility (e.g. being able to give special meaning to the distinction between unset and null) and convenience (e.g. allowing module authors to concisely express that something should never be null and be freed from using coalesce inline), so I too would love to hear from you all about which of these designs would be most useful in the modules you’re hoping to use optional attributes in.
Thanks again, both @alisdair and @apparentlymart! I also think the current behavior is intuitive. But that may be because it matches the current behavior of variables, and I have a little too much experience with terraform to be entirely objective about what is intuitive or not. And I also tend towards enabling advanced use cases over always keeping things simple, anyways.
I do rather like having the option of the current behavior, as it does allow me to set a default, but also create a special condition when the value is null and do something else.
But that does open up the nullable question again, as I am sure someone will want to prevent this behavior. Perhaps a third argument to optional(<type>, <default>, <nullable>)?
I must admit I do feel loathe to replicate the “we don’t know so I guess we’ll allow both” path that we ended up on with input variables, but if we can see benefits to either side that outweigh the considerable complexity in there being two possible modes for users to understand and decide between then that is indeed a possible third option.
If we do make it something the module author gets to decide then I would suggest making it be non-nullable by default and have nullability be the explicit opt-in, even though that is the opposite of how variables work, because when we added nullable = false for variables we did so with the intention that false would become the default value for that argument in a later language edition, assuming we eventually accumulate enough language design regrets to justify one.
I suppose that while I am greatly looking forward to this optional() function in tf 1.3, I still wish objects might be implemented in a way where they are just “nested variables” such that each attribute supports all the same arguments as the variable block.
variable "foo" {
type = object({
variable "a" {
type = string
default = "hello"
description = "just a string"
nullable = false
}
})
}
Yeah, it’s super verbose, but it’s complete. And having descriptions is great for documentation.
Hi again all!
As you can see in the chatter above, @lorengordon’s question about our handling of null values made us realize that our team didn’t have a shared understanding of what we’d designed and what was implemented.
This is a great example of why we like to share work in progress in these alpha releases and react to your questions and feedback, so thanks for drawing attention to that!
Since then we’ve been reviewing some of the use-cases that motivated this feature, and reviewing similar behaviors elsewhere in Terraform. Based on that we’ve made a small design change in today’s v1.3.0-alpha20220817 release which will make Terraform treat a null value in the way I described in my earlier incorrect comment.
Specifically, if an attribute has a default value specified then Terraform will use that default value both when the attribute is entirely omitted by the caller and when the caller explicitly sets it to null.
We’ve made this change to support the established patterns for dynamically deciding to leave an optional attribute unset. The simplest form of that is a conditional expression where one of the result arms is null:
example = {
foo = var.predicate ? "foo" : null
}
There are other similar situations that don’t involve conditionals, but the above is the most common technique.
The general idea is to make the two mechanisms involved here work independently of one another:
- An optional attribute means that Terraform will synthesize a null attribute of the appropriate type when converting a source object that omits that attribute.
- Specifying a default means that Terraform will replace a null value with the given default value after type conversion is complete, regardless of why the attribute turned out to be null.
This then matches how Terraform treats optional arguments in resource blocks: Terraform Core handles the conversion to the given object type and then the provider replaces any null value with its default value.
We’ve done our best to try to identify potential real-world situations where it would be important to distinguish an explicit null value from an null value implied by omission, but we didn’t find any compelling examples in our analysis so far.
We’d love to hear from you if you try the new behavior in the latest alpha and find any specific real-world situations where being unable to recognize this difference is problematic.
In response to any examples we will first try to imagine a different way for you to model the situation using this updated design. If we cannot find a reasonable answer then we still have some time to adjust the design further before including it in the final v1.3 release, or to convince ourselves that the need could be addressed by a backward-compatible addition in a later release.
This comment is already far too long so I won’t go into the details here, but we do have sketches for some different ways to extend the attribute declaration syntax with more features in a backward-compatible way, and so the door is still open for more detailed constraints in later versions.
Thanks again for trying this out and sharing your feedback!
Hmm… Personally, I feel like I end up using explicit null checks in modules all the time, to condition a resource or dynamic block. So my hot take, this feels undesirable, especially without a nullable equivalent. Workaround would require yet another attribute/argument, so the value remains separately testable. If I didn’t want null passed in at all, I think I would just set a validation condition, instead?
Hi @lorengordon,
Can you share a specific example of what you mean?
Note that this doesn’t prevent you from using optional without a default in order to let a plain null show through and make checks based on that. It only prevents you from writing logic that differentiates between the caller explicitly writing null vs. omitting the optional attribute altogether, because we intend for callers to understand those as exactly equivalent in the same way they are in resource blocks.
It’s a little hard to come up with an exact example because I don’t use experimental features. I have examples where I have an object attribute that I test explicitly for null, but I don’t bother setting a default for the entire object, and can’t yet set a default for just the attribute.
I do have an example where we use a condition on the variable instead of the object attribute. We set a default value, and also test whether the user passed null, using null as the condition to skip creation. This works because nullable = true by default. Without that, which is basically the change in question for object attributes, I don’t think this would work.
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
count = var.server_side_encryption_configuration == null ? 0 : 1
bucket = aws_s3_bucket.this.id
rule {
bucket_key_enabled = var.server_side_encryption_configuration.bucket_key_enabled
apply_server_side_encryption_by_default {
sse_algorithm = var.server_side_encryption_configuration.sse_algorithm
kms_master_key_id = var.server_side_encryption_configuration.kms_master_key_id
}
}
}
variable "server_side_encryption_configuration" {
description = "Schema object of the server side encryption configuration"
type = object({
bucket_key_enabled = bool
kms_master_key_id = string
sse_algorithm = string
})
default = {
bucket_key_enabled = true
kms_master_key_id = null
sse_algorithm = "aws:kms"
}
}
Hi @lorengordon,
I think what you are saying is that it isn’t clear how you would reproduce this server_side_encryption_configuration variable if it were a sub-attribute of another variable, rather than a top-level attribute itself.
To try to turn that into an example using these new features, I’m going to just guess that it might make sense for this variable to appear as a nested object inside a more general variable called s3, declared like this:
variable "s3" {
type = object({
server_side_encryption = object({
bucket_key_enabled = bool
kms_master_key_id = string
sse_algorithm = string
})
})
}
From studying your example I think you are saying that the important detail here is that you want server_side_encryption_configuration to be enabled by default, but you want to allow users of the module to explicitly turn it off by setting it to null, thereby overriding the default value to be null instead.
If so, then indeed that would not be possible under the new design here. Under the rules that we have for the rest of Terraform (ignoring the accidental special case for input variables), it would be necessary to use an explicit boolean value, or some other non-null value, to represent affirmatively disabling something.
Given that, an optional object types version of this design (and also a nullable = false version of this design) would need to follow the same design patterns a provider developer would follow when modelling something like this, which is to add an explicit enabled attribute:
variable "s3" {
type = object({
server_side_encryption = optional(
object({
enabled = optional(bool, true)
bucket_key_enabled = optional(bool)
kms_master_key_id = optional(string)
sse_algorithm = optional(string)
}),
{
bucket_key_enabled = true
sse_algorithm = "aws:kms"
},
)
})
validation {
# If enabled = false then the other options are forbidden
condition = (
!var.s3.server_side_encryption.enabled ? (
var.s3.server_side_encryption.bucket_key_enabled == null &&
var.s3.server_side_encryption.kms_master_key_id == null &&
var.s3.server_side_encryption.sse_algorithm == null
) : true
)
error_message = "Cannot set .server_side_encryption settings when enabled = false."
}
validation {
# If enabled = true then some other options are required
condition = (
var.s3.server_side_encryption.enabled ? (
var.s3.server_side_encryption.sse_algorithm != null
) : true
)
error_message = "Must set .server_side_encryption.sse_algorithm when enabled = true."
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
count = var.server_side_encryption_configuration.enabled ? 1 : 0
bucket = aws_s3_bucket.this.id
rule {
bucket_key_enabled = coalesce(var.server_side_encryption_configuration.bucket_key_enabled, true)
apply_server_side_encryption_by_default {
sse_algorithm = var.server_side_encryption_configuration.sse_algorithm
kms_master_key_id = var.server_side_encryption_configuration.kms_master_key_id
}
}
}
It’s notable that for the equivalent of this design in a provider schema the validation blocks in my example above would be written as logic in the provider’s implementation language (typically Go) rather than as expressions in the Terraform language:
if sse.enabled {
if sse.bucket_key_enabled != nil || sse.kms_master_key != nil || sse.sse_algorithm != nil {
// (return an error)
}
} else {
if sse.sse_algorithm == nil {
// (return an error)
}
}
So while this is essentially functionally equivalent to the corresponding design pattern for providers, I think it’s debatable how the readability/maintainability of it compares, given that Terraform module authors aren’t typically expected to deal with complex boolean expressions like those, whereas they are more common in a general-purpose programming language.
With that all said, I think it’s worth noting that the original design of treating null as special (without nullable = false set) does prevent the conditional-setting pattern that this design change was intended to enable.
Here’s a hypothetical module call using the variable as @lorengordon originally presented it, rather than my modified example:
module "s3" {
# ...
server_side_encryption = (
var.kms_master_key != null ?
{
kms_master_key = var.kms_master_key
} :
???
}
The ??? placeholder in the above example represents “what can I write here to dynamically select the default value that the module author specified for this variable?” null would be our answer when nullable = false, but when nullable = true (the current default, arguably a mistake) the only way to get the default is to not specify the server_side_encryption argument at all, which prevents making a dynamic decision about it.
@apparentlymart I think you got it exactly.
And yes, for this use case we could use a new attribute specifically for the condition, instead of testing the entire object for null. It seems extraneous to me and requires more from the user, but that’s what I was trying to get at earlier:
Workaround would require yet another attribute/argument, so the value remains separately testable.
Thanks for confirming, @lorengordon!
The explicit extra argument is in line with our design expectations elsewhere in Terraform, so I expect that we will move forward with that as the recommendation for now and then in later releases we will consider possible improvements to the ergonomics of that design pattern, so that it’s possible to achieve an interface like what I showed without so much boilerplate.
The typical rule in the Terraform language is that null represents the absense of a value rather than being treated as a value in its own right, similar to its role in SQL. In other words, null is intended to represent “I have nothing to say about this attribute” rather than “I want to explicitly disable this attribute”. Given all of the existing design precedent in that direction and that input variables behaving differently was an accidental design mistake, I expect that our direction here will be towards making it more convenient to employ explicit mechanisms for disabling rather than towards making it easier to overload null to represent disabling in certain cases.
Reading through the examples provided in response to my use case, I am also struggling to see how the module author might set their own default, and also allow the user to pass null.
In this case, when the user passes null, it is intended to represent the absence of something, so they get the default implemented by the resource provider, instead of the default implemented by the module author.
Here is the design case equivalent to nullable = true. If the user passes null explicitly for any object attribute (say, bucket_key_enabled = null) then that value gets passed to the underlying resource argument and picks up the default value from the provider. Clean and easy.
variable "s3" {
type = object({
server_side_encryption = optional(
object({
bucket_key_enabled = optional(bool, true)
kms_master_key_id = optional(string)
sse_algorithm = optional(string, "aws:kms")
}),
{
bucket_key_enabled = true
sse_algorithm = "aws:kms"
},
})
)
}
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
count = var.s3.server_side_encryption_configuration == null ? 0 : 1
bucket = aws_s3_bucket.this.id
rule {
bucket_key_enabled = var.s3.server_side_encryption_configuration.bucket_key_enabled
apply_server_side_encryption_by_default {
sse_algorithm = var.s3.server_side_encryption_configuration.sse_algorithm
kms_master_key_id = var.s3.server_side_encryption_configuration.kms_master_key_id
}
}
}
Now I’m trying to imagine how to do the same for the design equivalent to nullable = false… I actually don’t think it’s possible? If the user ever passes null then the module-level default is assigned instead of the resource-level default, right? What am I missing? Is there another design pattern we’re supposed to be using?
variable "s3" {
type = object({
server_side_encryption = optional(
object({
enabled = optional(bool, true)
bucket_key_enabled = optional(bool, true)
kms_master_key_id = optional(string)
sse_algorithm = optional(string, "aws:kms")
}),
{
bucket_key_enabled = true
sse_algorithm = "aws:kms"
},
)
})
}
resource "aws_s3_bucket_server_side_encryption_configuration" "this" {
count = var.s3.server_side_encryption_configuration.enabled ? 1 : 0
bucket = aws_s3_bucket.this.id
rule {
bucket_key_enabled = ??? # How do I get the `null` from the user input?
apply_server_side_encryption_by_default {
sse_algorithm = var.s3.server_side_encryption_configuration.sse_algorithm
kms_master_key_id = var.s3.server_side_encryption_configuration.kms_master_key_id
}
}
}
(Note the ??? in the bucket_key_enabled expression. I am looking for guidance there.)