Dealing with unordered sets of complex objects / SetNestedAttribute with Computed attributes

Terraform Providers Plugin Development
1

Hi all,

While developing a Terraform Provider for our API, we stumbled upon the problem that our API has several attributes where the order is not defined. The API can return items in an order that differs from what is in the configuration. Our initial implementation used ListNestedAttribute, which obviously does not work when order changes. We then decided to switch to SetNestedAttribute, but we did not realize that this actually makes the problem worse. The list could result in mismatches due to changes in order. The set however gets confused all over the place, because our elements also contain Computed attributes, which can change at every request. This causes the set to completely loose track of its elements, resulting in plans which essentially try to recreate the contents of every set every time.

I’ve done some searching on this forum and found several other users with the same issue, but none with a solution:
SetNestedAttribute with nested computed attributes
Explicit hash function for SetNestedAttribute
Framework v1.1.1 - Strategies for diagnosing resource state churn

Somewhere, I found a suggestion to use a map. This might work for nested elements that define a single unique key that a user knows upfront and that needs to be put in the config anyway, but many of our nested elements only define an implicit identifier that is generated when the element is created. This rules out a map for our case.

With list, set and map all eliminated, I’m out of options. So my question is: how do you deal with unordered lists? At Custom sort for ListNestedAttributes the suggestion was made to sort the elements. I can see how I might sort the elements in Read, Create and Update methods to match the state, but I don’t understand the part about the PlanModifier. Why is that needed? And how would you only discard changes that result from differences in ordering, without discarding any other potential changes? Maybe somebody already implemented this approach somewhere and could point to an example?

Btw, the provider I’m working on is here: GitHub - topicuskeyhub/terraform-provider-keyhub . However, the code is difficult to read, because it is entirely generated from an OpenAPI specification and testing it requires access to a working Topicus KeyHub deployment. The generated documentation does however describe the structure of some of these troublesome attributes, such as the account and client_permissions attributes for the keyhub_group resource: Terraform Registry

Best regards,
Emond

2

Hi @papegaaij,

That combination of unordered objects with partially computed values can make it very difficult to map to the declarative configuration of Terraform. There needs to be some way for Terraform to correlate old and new values to present a diff of individual values, but when you have a set, there is no way to do this. This doesn’t necessarily mean the set is not working correctly (it should work just fine), but the diffs presented in the plan won’t be as fined grained as you would like.

The only way to have the plan show detailed changes to each individual value, is for Terraform to have a way to correlate those values, either via the order in a list, or the key in a map.

Trying to maintain a sorted list mostly works, but the problem is that during the read call you can’t match any ordering within the configuration, which will cause drift when refreshing the resource. This is where the plan modifier would come in, to accept a difference in order as semantically equal when there are no other changes.

Another approach may be to split the data into multiple data structures, making the config more like the “input” and the computed values like an “output”. By putting all the user definable attributes in one structure, and segmenting out the computed attributes into another with a defined way to map between them, it can make it easier to control the plans that users see. This may also help with the attributes which change on every read too, as they are not particularly useful in Terraform, rather they are a hinderance to converging on a stable state. They might still be useful from an informational stand point, and keeping them segregated reduces the chance of them interfering in the diff rendering.

3

Hi @jbardin ,

Thanks for your comprehensive reply. It would be really nice if Terraform would define a way to specify the identifying attributes for nested objects. That would allow Terraform to locate the correct elements in the set, even if the values do not entirely match. If not specified, all attributes would used, like it works at the moment. This would work both on the provider and the Terraform side, but probably require a change in the protocol. Something like this:

resp.Schema = schema.Schema{
    Attributes: map[string]schema.Attribute{
        "items": schema.SetNestedAttribute{
            Required: true,
            IdentifyingAttributes: []string{"id"},

            NestedObject: schema.NestedAttributeObject{
                Attributes: map[string]schema.Attribute{
                    "id": schema.Int64Attribute{
                       Required: true,
                    },
                    "last_used": schema.StringAttribute{
                        Computed: true,
                    },
                },
            },
        },
    },
}

For now, I think we will go for the sorted list approach. I see the problem with read now. Am I correct in that this would only be a problem if the state does contain the set, for example after an import? I’m still quite new to provider development. This is how I understand it works:

  • Create gets a Plan that is constructed from the config. In this Plan, the items have a specific order, and Create should make sure it constructs a State with the elements in the same order.
  • Read gets the current State, which should be in the order of the elements as returned by Create, and thus in the same order as in the config.
  • If Read gets called as part of an import, there is no real prior state and there is no way to return the elements in the same order as in the config. This is where the PlanModifier comes into play.

Do you know an example that implements such a PlanModifier? I’ve no experience with those other than using the predefined modifiers, such as UseStateForUnknown and RequiresReplace. I guess this modifier can be quite hard to get right, because it has to deal changes in ordering, but still keep actual changes (for example two elements gets swapped, but on one of the two a configured attributes has also changed).

I guess, this also seems like another legitimate reason to address Add `Config` to the `resource.ReadRequest` · Issue #878 · hashicorp/terraform-plugin-framework · GitHub.

Splitting the elements into two parts will be very hard for us to get right. Not only will it require massive changes in the way to code is generated, but I don’t see a good way to allow the user to relate a configured element to the one containing the computed attributes. This effectively puts the burden of locating the elements in the set on the user. It would be feasible for us to move the computed elements into a SingleNestedElement attribute inside the element itself, but that does not solve the issue, because there is no way to ignore this attribute in the comparison.

Best regards,
Emond

4

It would be really nice if Terraform would define a way to specify the identifying attributes for nested objects

That’s something which has been considered, both for resources as a whole and nested objects (we may still add something like that for resources). It doesn’t solve the entire problem however, since identifying attributes are often computed, so during the plan they may be unknown giving nothing to compare.

  • Create gets a Plan that is constructed from the config. In this Plan, the items have a specific order, and Create should make sure it constructs a State with the elements in the same order.

Yes, a create plan cannot alter configured attributes, so it must always return the values in the same order.

  • Read gets the current State, which should be in the order of the elements as returned by Create, and thus in the same order as in the config.

Read cannot compare to the configuration, but you can however compare it to the prior state (which should reflect configuration in most cases).

  • If Read gets called as part of an import, there is no real prior state and there is no way to return the elements in the same order as in the config. This is where the PlanModifier comes into play.

Exactly, Read may have inadvertently re-ordered things, so the plan modifier need to check for semantic equality when there are no changes (a plan cannot alter a config value, but it can choose the prior state if it determines it makes no changes). Once there is a change caused by the config, then the items will be re-ordered again in state.

I have not worked much with provider implementations, but you might be able to get some ideas from the semantic equality additions in the framework, starting with this PR: Type-Based Semantic Equality by bflad · Pull Request #737 · hashicorp/terraform-plugin-framework · GitHub (docs: Plugin Development - Framework: Handling Data - Custom Types | Terraform | HashiCorp Developer)

It’s unlikely that the configuration data will be added directly to the Read call (for the same reason it was rejected from the Import call), unless we have a situation which absolutely cannot be solved in another way. The Read calls are intended to allow Terraform to get a representation of the real-world state of the resource, which should not depend at all on the configuration. Adding configuration further muddles the concepts of prior state and desired state, and encourages implementations to try and subvert the changes Terraform is trying to detect.

5

That’s something which has been considered, both for resources as a whole and nested objects (we may still add something like that for resources). It doesn’t solve the entire problem however, since identifying attributes are often computed, so during the plan they may be unknown giving nothing to compare.

Yes, I understand, the corner cases are always difficult to get right. I think at the moment the unknown values are simply ignored. A similar approach could also work on a restricted set of attributes. Of course, if all those attributes are unknown, there’s no way to correlate the elements. But this is also the case in the current situation: if all attributes of a nested element are unknown, there’s nothing to compare. This probably does not happen a lot, but I think it still is a valid case.

Exactly, Read may have inadvertently re-ordered things, so the plan modifier need to check for semantic equality when there are no changes (a plan cannot alter a config value, but it can choose the prior state if it determines it makes no changes). Once there is a change caused by the config, then the items will be re-ordered again in state.

This indeed is a difficult case. If a Read does not have a prior state, and returns the elements in the wrong order (i.e. the order does not match the config), it’s very hard to get it right again. Refreshing does not work, because Read will get the prior state, which is in the wrong order. For now, I think I’ll start with the reordering in Create and Read and leave the PlanModifier for later. We still have some issues with import anyway because we cannot access the config.

This is not related to the unordered lists, but the absence of config in Read is giving us trouble determining what must be read. We only fetch the attributes that are configured. This works fine via Create, but when using Import, we do not know which attributes are configured. This causes the provider not to fetch any attributes at all, resulting in state mismatches all over the place. We can fix this by requiring the user to enter all attributes when using import, but this is not very user friendly and also error prone.

Best regards,
Emond

6

Yeah, the need for some sort of configuration for import is apparent, and something we intend to look into more closely as we add features to the import mechanism. The actual resource configuration was rejected as I mentioned, but there may end up being something like an “import config” to bootstrap the process, help with discovery, and config generation.

7

I did some testing and implemented the reordering of the state as part of Create/Read/Update to make it match the prior state. This works fine as long as the prior state and config define the same order. However, if I change the order of the config, the reordering uses the prior state as baseline, resulting in a mismatch in the order and an in-place update for the resource (as expected). So, I went ahead and tried to implement the PlanModifier, and got stranded.

My current PlanModifier is very simple and is only the test the basics. It only deals with accounts that are member of a group (simplified):

resource "keyhub_group" "terra" {
  name = "Terraform"
  accounts = [{
    uuid   = "2948741d-f852-4599-be0e-cf187b306b4b"
  },{
    uuid   = "7ea6622b-f9d2-4e52-a799-217b26f88376"
  }]
}

The implementation of the PlanModifier swaps the accounts in the plan if the uuid of the first configured account does not match that of the state:

config0 := req.ConfigValue.Elements()[0].(types.Object).Attributes()["uuid"]
state0 := req.StateValue.Elements()[0].(types.Object).Attributes()["uuid"]
if !config0.Equal(state0) {
    vals := req.PlanValue.Elements()
    resp.PlanValue = types.ListValueMust(
        resp.PlanValue.ElementType(ctx), []attr.Value{vals[1], vals[0]})
}

This eliminates the account uuids from the plan, but the plan still contains a lot of other changes:

  # keyhub_group.terra will be updated in-place
  ~ resource "keyhub_group" "terra" {
      ~ accounts                     = [
          ~ {
              + directory_uuid           = (known after apply)
              ~ disconnected_nested      = false -> (known after apply)
              ~ last_used                = "2024-01-17" -> (known after apply)
              ~ nested                   = false -> (known after apply)
              ~ provisioning_end_time    = "2024-01-17T14:41:20.802612Z" -> (known after apply)
              ~ two_factor_status        = "TOTP" -> (known after apply)
              ~ visible_for_provisioning = true -> (known after apply)
                # (2 unchanged attributes hidden)
            },
          ~ {
              + directory_uuid           = (known after apply)
              ~ disconnected_nested      = false -> (known after apply)
              ~ last_used                = "2024-01-17" -> (known after apply)
              ~ nested                   = false -> (known after apply)
              ~ provisioning_end_time    = "2024-01-17T14:41:20.792113Z" -> (known after apply)
              ~ two_factor_status        = "WEBAUTHN" -> (known after apply)
              ~ visible_for_provisioning = true -> (known after apply)
                # (2 unchanged attributes hidden)
            },
        ]
        name                         = "Terraform"
      + administered_clients         = (known after apply)
      + administered_systems         = (known after apply)
      ~ audit_requested              = false -> (known after apply)
      ~ auditor                      = false -> (known after apply)
      + authorized_groups            = (known after apply)
      ~ authorizing_group_types      = [] -> (known after apply)
        ... many more
        # (14 unchanged attributes hidden)
    }

I also tried replacing resp.PlanValue with req.StateValue for accounts. That does eliminate the changes for that attribute, but all other changes remain. I can add UseStateForUnknown() on all computed attributes, but that seems like a hackish solution. What’s the best way to solve this?