Suggestions for Overriding Module Attribute

rpattcorner · January 11, 2023, 11:55pm

I’m having difficulty locating a technique to add an optional attribute to a child module invocation. The child module is our own and under our control. The idea is that generally the module will be invoked without the attribute:

module "foo" {
  source      = "../modules/foo"
 this = "foo1"
  providers = {
    aws.one = aws.one
    aws.two   = aws.two
  }
}

But there are times we need to pass an overriding attribute which the module will act upon:

module "foo" {
  source      = "../modules/foo"
 this = "foo1"
 optional_attribute = "fee"
  providers = {
    aws.one = aws.one
    aws.two   = aws.two
  }
}

The child module invocation is shared by symlink to a number of root modules. Obviously we can solve the problem by providing a non-symlinked version in the root module, but it would be great to have the attribute be sent when a value for it is present and skipped when a value for it is absent.

Our initial approach was to use a variable as the source of the optional attribute, but we don’t want to declare an occasionally used root module attribute in every single root module when it’s used very seldom.

However it looks like there is no way to test if a variable exists and act accordingly, e.g. send the module a null that it can test or a value. At least one bit of documentation suggests that such an approach goes against the grain of a variable, as they “must” exist in some form.

We can however possibly get the value out of state, but again have not found a technique.

Here are some of the things we’ve tried and failed with (each represents an approach, we’d only use one):

  #optional_attribute = var.optional_attribute 
  #optional_attribute  = try(var.optional_attribute, true) ? var.optional_attribute : null
  #optional_attribute = try(var.optional_attribute, null)
  #optional_attribute = local.account-map.optional_attribute = null ? "" : local.account-map.optional_attribute
  #optional_attribute = can(local.account-map.optional_attribute) ? local.account-map.optional_attribute : null

Any ideas?

jbardin · January 12, 2023, 2:03pm

Hi @rpattcorner,

Much like a compiled programming language, all symbols in Terraform configuration must be statically defined in the source, so there’s no way to conditionally define a variable.

Assigning a value to a module variable however is optional if it has a default value, and null is a perfectly valid default. You can simply assign that variable as necessary within the module, and leave it as null when it is unused.

rpattcorner · January 12, 2023, 3:09pm

Thanks. Yes, what I’m seeking is a little different … a statement to test whether a value is defined or not.

We have a large number of independent Terraform runs that invoke the child module, and only a small fraction of them require the optional variable. So looking for a way to test whether that variable or value exists in any particular independent invocation.

I say variable or value because it looks like it might be allowed to test a value as present or absent in a map in state where it might not be allowed to test a variable as declared or not.

jbardin · January 12, 2023, 3:36pm

I’m not quite sure I follow what you’re trying to do here. A variable being used at all must always be declared for the configuration to be valid, there is no way to test whether a variable is declared at runtime. If you are looking to conditionally have a value, you could check for its existence in a map, but you would need a variable to store the map, so there is nothing gained there if this map does not already exist.

rpattcorner · January 12, 2023, 3:43pm

Well, we may be close then. There is a map, always available in the state of the individual TF run, and the map either does or does not have an entry for our optional_attribute.

Up to now (formerly running on 0.13) we had to define the optional_attribute in every single instance of the map in state, but with the ongoing migration to 1.3.7 I believe we can now define the optional_attribute entry in the map as an optional variable which means we could (in the tf code that creates the state) only define optional attribute values in those maps that actually need the optional attribute.

I hope.

2 questions on that approach since I can’t seem to get it to work:

Does the TF code that creates the state need to define the optional_attribute element of the map as having a null default as well as being optional?
What expression in the TF code that reads the state can determine if the optional_attribute has been set? Are we talking about testing it for a null value?

jbardin · January 12, 2023, 3:57pm

I’m really having a hard time understanding what it is you’re trying to do here; a minimal, reproducible example would help a lot.

Maps don’t have attributes, so there is no way to optionally define one. Any key-value pair in a map is essentially optional. An object can have optional attributes, and a module can have optional input variables.
I don’t know exactly what you mean by reading the state, but you can conditionally get a map element using the lookup function or the try function.

rpattcorner · January 12, 2023, 4:00pm

Understood … I was trying to avoid an example because it’s complicated … one TF run establishes values in a shared state that a series of independent other TF runs reads. But first let me have a go at lookup(), which might do exactly what’s needed

apparentlymart · January 12, 2023, 4:13pm

I think there might be some terminology confusion here based on your original message: you are talking about “attributes” but the example code you showed was illustrating an input variable called optional_attribute. There are similar capabilities for whole input variables vs. attributes of an object given in an input variable but they are subtly different so good to be clear about which you are talking about.

If you are familiar with general-purpose programming languages then it might help to think of the input variables (declared by the presence of a variable block in your child module) as similar to arguments to a function. Some function arguments can be of an object type that has nested attributes inside it, and in Terraform that is represented by a variable having an object type as its type constraint.

The other thing to be aware of here is the distinction between declaration and definition. The variable block and its type constraint declare the variable, stating that the variable exists and can be defined by the caller of the module.

The definition is what your caller writes in the module block when calling your module. All variables and attributes that are declared inside the module always exist regardless of what the caller writes, but you can write a declaration that will cause Terraform to automatically provide certain values when the caller doesn’t set them, instead of raising an error:

For whole input variables:

If you set default inside the variable block then Terraform will automatically set the variable to that value when the caller doesn’t explicitly define it. Terraform is essentially quietly adding a hidden definition of the variable, because all variables must always be defined even if some definitions are automatic.
You can set default = null to specify that the default definition should be null. null is how Terraform represents “no value”, so in this case the variable is being automatically defined as having no value, but the variable still exists because it is still declared inside your module.

The possibilities for object type attributes are similar but they have a different syntax due to being embedded inside a type constraint and therefore needing to fit within the type constraint syntax:

If you declare an attribute as optional(string, "hello") then during type conversion Terraform will behave as if the caller had defined this attribute as "hello" unless the caller explicitly set it.
If you use the single-argument form of optional then the default value is null, again representing “no value”.

The presence or absence of a variable or object attribute is fixed as part of its declaration and so cannot vary dynamically at runtime. But you can define the variable or attribute as null to represent that it has no value assigned to it in a particular case; that is how we represent the absence of a value dynamically at runtime.

rpattcorner · January 12, 2023, 4:24pm

That’s really helpful, thanks!

In the current context:

It seems clear that there is no test to determine if a variable of a given name has been defined. Too bad, but so be it.
So we have object attributes. This looks promising

The object in question lives in state and is created by a higher level TF run that creates shared state that all the independent other TF runs read. So variables and their setting (via the higher level TF terraform.tfvars) are involved, but not directly … the end result of the higher level TF run is readable state for the other runs.

From what I’m reading here, it sounds like the way to go is:
In the higher level TF:

Create an optional attribute with a default null (single value argument, or specify)
In ‘terraform.tfvars’ override that null by setting a value in the occasional case where we need our optional attribute to be acted upon

In the lower level TF runs:

Get the object placed by the high level run from shared state
See if optional_attribute exists in the object via the lookup() function
If it exists pass its value, if not pass a null

Or perhaps more simply, pass its value … period. If I understand you right the child module, when passed a null, can detect the null as a value and decline to act on the optional attribute

Does that sound about right?

apparentlymart · January 13, 2023, 1:41am

From Terraform’s perspective all declared input variables and object attributes are always defined, but sometimes they are defined automatically based on defaults and other times they are defined explicitly by the caller.

You’re right that Terraform does not typically offer ways to distinguish between those two cases, and that’s intentional because sometimes callers need to decide dynamically whether to explicitly define a input variable or attribute, and so will write something like this:

  example = var.condition ? "example value" : null

I think what you went on to describe is what I would recommend: don’t try to distinguish between explicitly-defined and implicitly-defined at all and just let null mean what it’s intended to mean: there is no value, whether because the caller didn’t say anything about it and Terraform used the default or because the caller explicitly set the value to null (perhaps as one result of a conditional expression as I showed earlier).

The lookup function is intended for retrieving elements from map values rather than attributes from object-typed values. Because Terraform infers automatic conversions from object to map in most cases, you can use lookup with object attributes if you want. But if you do so you’ll need to be careful that lookup in that case effectively tests whether the attribute is defined as part of the type, not whether it has a value.

Concretely, if you have an object in var.example with an attribute foo that has been defined as null and you use lookup(var.example, "foo", "baz") then the result will be null rather than "baz" because the attribute foo is part of the object type of var.example and so it’s always defined, even though sometimes it’s defined to have no value (be null). lookup would only return "baz" here if the type constraint for var.example didn’t declare an attribute foo at all.

If you want to make dynamic decisions about based on whether object attributes have a value, the main intended methods are:

Directly comparing to null, like var.example.foo != null or var.example.foo == null.
Use the coalesce function, like coalesce(var.example.foo, "baz") which will return the first argument if it isn’t null, or the second argument otherwise.

These work with the dynamic value of the attribute rather than with the type of the containing object.

There is a small loophole for input variables in particular due to a historical design mistake in earlier versions of Terraform. It doesn’t exactly allow you to distinguish between explicitly or implicitly defined but it’s close enough to work if you really want to do it, although I would typically recommend against it because it would be surprising to folks expecting typical Terraform module behavior:

variable "example" {
  type     = string
  default  = "VERY UNLIKELY VALUE THAT NOBODY WOULD EVER SET DIRECTLY"

  # The following is actually the default but I'm
  # including it just to help explain what's going
  # on.
  nullable = true
}

Now if the caller of the module sets example = null then – because nullable = true – var.example will actually be null inside the module, rather than using the default value. Therefore you can approximate recognizing that situation by comparing to the silly default value that is very unlikely to actually be used:

var.example != "VERY UNLIKELY VALUE THAT NOBODY WOULD EVER SET DIRECTLY"

I’m mentioning this only because you seem interested in understanding the details of how Terraform treats these situations, but I would not recommend doing this except as a last resort.

If we had the opportunity to start fresh with Terraform today we’d make nullable = false the default and probably not offer the weird nullable mode at all, but this weird treatment of explicitly-setting null is a concession to backward compatibility due to the Terraform v1.x Compatibility Promises. Modules that don’t set nullable = false have surprising treatment of nulls that isn’t compatible with the conditional-null pattern described earlier.

rpattcorner · January 13, 2023, 4:45pm

Many thanks for the thoughtful reply! Seems to me the most straightforward approach is the one we’re both converging on , which is:

In the state creating tf run:

Define the optional object attribute with a null default
Occasionally override the default with a null value in terraform.tfvars, otherwise omit it from terraform.tfvars and let the null default take effect

In any of the state consuming TF runs:

Simply test the value for null. This means it was not overridden in the prior process tfvars
act accordingly

The only hitch is that it’s less than obvious how to set defaults on attributes of an object in a map. Is this even possible?

variable "example_map" {
  type = map(object({
    ordinary_attribute   = string
    other_ordinary       = string
    optional_attribute   = string
    yet_another_ordinary = list(string)
  }))

  default = {
    object ({
      optional_attribute = null
    })
}

with a tfvars like:

example_map = {
  "example_object_defaulting_optional" = {
      "ordinary_attribute" = "one"
      "other_ordinary"        = "two"
      # No optional attribute in this object, we want its value to default to null
      "yet_another_ordinary" = "three"
   }, # End example object defaulting optional value to null
  "example_object_setting_optional_att_value" = {
      "ordinary_attribute" = "one"
      "other_ordinary"        = "two"
      # Optional attribute not null in this rare instance
      "optional_attribute" = "NonNullValueGoesHere"
      "yet_another_ordinary" = "three"
   } # End example object setting optional value to something nonnull
}  # end map

This fails with a message that strongly implies that there’s not a general way to default an attribute in a map of objects, and that TF wants a complete definition of each object in defaults, not, as desired, a definition of the object’s default value in one case:

╷
│ Error: Missing attribute value
│ 
│   on var2.tf line 12, in variable "example_map":
│    9:   default = {
│   10:     object ({
│   11:       optional_attribute = null
│   12:     })
│   13:   }
│ 
│ Expected an attribute value, introduced by an equals sign ("=").

as shown here – clearly a nonstarter for what’s needed.

Is there another and better way to define a default (of null) in a definition of an attribute of an object in a way that does not require manually creating a value setting for the optional attribute in every single object in the map, or even worse, defining every object up front in variable defaults, including the optional, which amounts to the same thing.

“All” we are trying to do here is:

Be able to test the optional_attribute_value from retrieved state for nullness (or any other value)
In a way that does not require adding that optional_attribute_value to every single object in the map.

I realize in miniature case it’s a small savings in typing, but as there will be many objects and multiple optional attributes, eliminating a requirement to assign a value to every attribute in every object will eliminate a significant source of human error and make us far DRY’er.

stuart-c · January 13, 2023, 7:16pm

You need to be looking at optional attributes: Type Constraints - Configuration Language | Terraform | HashiCorp Developer

apparentlymart · January 13, 2023, 7:29pm

Indeed, the subtlety here is that the thing you apply the default to is the only thing whose nullness will be considered when deciding whether to insert the default.

If you define a default for the whole variable then that default is for situations where the whole variable is unset.

If you define a default for a particular attribute (using the optional attributes syntax) then it is the nullness of that particular attribute that will decide.

The other interesting thing to know here is that the defaults can nest: if you have an optional variable whose type contains an optional object attribute and then the default value of the entire variable is an object with that attribute unset, the default value for the attribute effectively becomes part of the default value for the entire variable too, because the default value for the variable gets converted to the variable’s declared type in exactly the same way as a value defined explicitly by the caller would be.

rpattcorner · January 13, 2023, 9:08pm

This is really excellent, appreciation all around, incl. @stuart-c for the optional attribute syntax I was somehow missing.

This looks right in test, will know more after Dr. King’s holiday when back at work.

For overall knowledge, here’s what seems to work:

In the variables:

variable "example_map" {
  type = map(object({
    ordinary_attribute   = string
    other_ordinary       = string
    optional_attribute   = optional(string, null)
    yet_another_ordinary = list(string)
  }))
}

In terraform.tfvars:

example_map = {
  "example_object_defaulting_optional" = {
    "ordinary_attribute" = "one"
    "other_ordinary"     = "two"
    # No optional attribute in this object, we want its value to default to null
    "yet_another_ordinary" = ["three"]
  }, # End example object defaulting optional value to null
  "example_object_setting_optional_att_value" = {
    "ordinary_attribute" = "one"
    "other_ordinary"     = "two"
    # Optional attribute not null in this rare instance
    "optional_attribute"   = "NonNullValueGoesHere"
    "yet_another_ordinary" = ["three", "four"]
  } # End example object setting optional value to something nonnull
}   # end map

And the great-looking-so-far result which looks just like what is needed:

$ tfnew console
> var.example_map
tomap({
  "example_object_defaulting_optional" = {
    "optional_attribute" = tostring(null)
    "ordinary_attribute" = "one"
    "other_ordinary" = "two"
    "yet_another_ordinary" = tolist([
      "three",
    ])
  }
  "example_object_setting_optional_att_value" = {
    "optional_attribute" = "NonNullValueGoesHere"
    "ordinary_attribute" = "one"
    "other_ordinary" = "two"
    "yet_another_ordinary" = tolist([
      "three",
      "four",
    ])
  }
})
> exit

(love that console! Too bad hard to use in modules …)

Eminently testable for nulls!