Framework v1.1.1 - Strategies for diagnosing resource state churn

I’m stuck diagnosing constant resource state churn with a SetNested Object in framework v1.1.1.

My first thought was to compare the state object retrieved with req.State.Get() against the final result I set with resp.State.Set().

I’m doing that by calling ToTerraformValue() on each and then logging the results of their String() methods. Maybe there’s a better way to do comparison? In any event… They’re the same string.

Next I decided to just re-use the state object retrieved from the Terraform backend without ever talking to the API.

My Read() method is now reduced to:

func (o *resourceFoo) Read(ctx context.Context, req resource.ReadRequest, resp *resource.ReadResponse) {
	// Retrieve state from TF backend.
	var state foo
	resp.Diagnostics.Append(req.State.Get(ctx, &state)...)
	if resp.Diagnostics.HasError() {
		return
	}

	// Short circuit! Send that state right back where it came from.
	resp.Diagnostics.Append(resp.State.Set(ctx, &state)...)
	return
}

Same result: mystery state changes are produced every time Read() runs.

I expected this experiment would always produce a “No changes” condition. Either I need to adjust my understanding of things, or something’s wrong.

Should this Read() function ever produce state churn?

Thanks!

edit: I’m beginning to wonder if #644 is related.

The SetNested attribute that’s giving me trouble has a Computed ListNested attribute within and it looks like that ListNested attribute is the one that’s invalidating the set member. Further, I’m using the listplanmodifier.UseStateForUnknown(), just like in #644.

Hi @hQnVyLRx :wave: Thank you for raising this topic and hopefully we can get this sorted for you.

The code snipped you show there is directly copying the prior state into the refreshed state. If everything in the terraform-plugin-framework and underlying terraform-plugin-go type systems is working as expected, there should be no “differences” between the two. I’m using quotes because by definition, certain types, such as maps and sets, are unordered so it is considered valid for there to be ordering changes within those.

Which brings me to asking a clarification question – could you explain more about your situation? I see you mention “resource state churn with a SetNested Object”, but what exactly are you seeing (Terraform output?) and what Terraform operations are involved? From there, I think we will be able to better able to narrow down what unexpected behaviors you are encountering and why they may be happening.


For what its worth, there have been some very recent Terraform 1.4 slated changes with how plans are calculated with sets, such as a new method of `ProposedNew` set comparison by jbardin · Pull Request #32563 · hashicorp/terraform · GitHub. If you’re able to run your configuration against a main branch build of Terraform, there may be material differences.

The main branch of terraform-plugin-framework also has internal/fwserver: Log detected plan value differences before unknown marking by bflad · Pull Request #630 · hashicorp/terraform-plugin-framework · GitHub, which may help troubleshooting unexpected plans being caused by other attribute handling.

Thanks, @bflad.

directly copying the prior state into the refreshed state … should be no “differences” between the two

This was my expectation.

My resource has a bunch of nested objects. It looks roughly like the following (many irrelevant attributes removed):

func (o *myResource) Schema(_ context.Context, _ resource.SchemaRequest, resp *resource.SchemaResponse) {
	resp.Schema = schema.Schema{
		Attributes: map[string]schema.Attribute{
			"aaa": schema.SetNestedAttribute{
				Required:      true,
				PlanModifiers: []planmodifier.Set{setplanmodifier.UseStateForUnknown()},
				NestedObject: schema.NestedAttributeObject{
					Attributes: map[string]schema.Attribute{
						"bbb": schema.SingleNestedAttribute{
							Computed:      true,
							PlanModifiers: []planmodifier.Object{objectplanmodifier.UseStateForUnknown()},
							Attributes: map[string]schema.Attribute{
								"ccc": schema.ListNestedAttribute{
									Computed:      true,
									PlanModifiers: []planmodifier.List{listplanmodifier.UseStateForUnknown()},
									NestedObject: schema.NestedAttributeObject{
										Attributes: map[string]schema.Attribute{
											"ddd_1": schema.Int64Attribute{
												MarkdownDescription: "Physical vertical dimension of the panel.",
												Computed:            true,
												PlanModifiers:       []planmodifier.Int64{int64planmodifier.UseStateForUnknown()},
											},
											"ddd_2": schema.Int64Attribute{
												Computed:      true,
												PlanModifiers: []planmodifier.Int64{int64planmodifier.UseStateForUnknown()},
											},
											"ddd_3": schema.ListNestedAttribute{
												Computed:      true,
												PlanModifiers: []planmodifier.List{listplanmodifier.UseStateForUnknown()},
												NestedObject: schema.NestedAttributeObject{
													Attributes: map[string]schema.Attribute{
														"eee_1": schema.Int64Attribute{
															Computed:      true,
															PlanModifiers: []planmodifier.Int64{int64planmodifier.UseStateForUnknown()},
														},
														"eee_2": schema.StringAttribute{
															Computed:      true,
															PlanModifiers: []planmodifier.String{stringplanmodifier.UseStateForUnknown()},
														},
														"eee_3": schema.SetAttribute{
															Computed:      true,
															ElementType:   types.StringType,
															PlanModifiers: []planmodifier.Set{setplanmodifier.UseStateForUnknown()},
														},
													},
												},
											},
										},
									},
								},
							},
						},
					},
				},
			},
		},
	}
}

When I first terraform apply, everything goes the way I expect: The resource is created and the Computed attributes are learned and saved to the Terraform state.

Things go sideways on Read(), and have continued to do so even with the short-circuit Read() function I posted previously.

The plan output indicates that set aaa is scheduled to be modified, replacing its only element with a new element. That new element is identical to the previous one (elements I’ve omitted here are all the same), except that its ccc is now an empty list.

The ccc list wasn’t empty in the object retrieved from, and then immediately re-committed to the state. I think something happened in plan generation.

The plan output looks like this:

  # foo.r will be updated in-place
  ~ resource "foo" "r" {
        id                         = "qnxzfxucqo-uyigprhs6zw"
      ~ aaa                        = [
          - { 
              - bbb               = {
                  - name   = "some_string" -> null
                  - ccc = [
                      - { 
                          - ddd_1 = 7 -> null
                          - ddd_2 = 1 -> null
                          - ddd_3 = [
                              - { 
                                  - eee_1 = 7 -> null
                                  - eee_2 = "10G" -> null
                                  - eee_3 = [
                                      - "val1",
                                      - "val2",
                                      - "val3",
                                    ] -> null
                                },
                            ]
                        },
                    ]
                }
              - name            = "some_other_string" -> null
              - some_identifier = "yet_another_string" -> null
            },
          + { 
              + bbb               = {
                  + name   = "name_bbb"
                  + ccc = [             <---------- this is not expected to be empty
                    ]
                }
              + name            = "some_other_string" 
              + some_identifier = "yet_another_string" 
            },
        ]
        name                       = "aaa terraform"
        # (2 unchanged attributes hidden)
    }

I’ve likely made some transcription errors here. I’m hoping to soon have a sanitized, simple provider which will make it easy to replicate the issue.

Hey @bflad,

I think I’ve made the problem easily reproducible. It boils down to:

  1. git clone
  2. go install
  3. terraform plan

The provider comes in under 200 lines in total, and half of it is the Schema() method.

repository here

It may not be related to #644, but that’s what I was thinking when I put this sample together.

@hQnVyLRx you can eliminate the Read call entirely from troubleshooting by using terraform plan -refresh=false. Building the provider with the latest changes on terraform-plugin-framework yields some key log entries:

2023-02-06T09:04:12.081-0500 [DEBUG] provider.terraform-provider-issue644: Detected value change between proposed new state and prior state: @caller=/Users/bflad/src/github.com/hashicorp/terraform-plugin-framework/internal/fwserver/server_planresourcechange.go:191 tf_resource_type=issue644_rack_type tf_rpc=PlanResourceChange @module=sdk.framework tf_attribute_path=leaf_switches tf_provider_addr=example.com/chrismarget/issue644 tf_req_id=38190b69-4c1a-a9bb-7420-8a503757b6ac timestamp=2023-02-06T09:04:12.081-0500
2023-02-06T09:04:12.081-0500 [DEBUG] provider.terraform-provider-issue644: Marking Computed attributes with null configuration values as unknown (known after apply) in the plan to prevent potential Terraform errors: tf_attribute_path=id tf_provider_addr=example.com/chrismarget/issue644 tf_req_id=38190b69-4c1a-a9bb-7420-8a503757b6ac @module=sdk.framework tf_rpc=PlanResourceChange @caller=/Users/bflad/src/github.com/hashicorp/terraform-plugin-framework/internal/fwserver/server_planresourcechange.go:200 tf_resource_type=issue644_rack_type timestamp=2023-02-06T09:04:12.081-0500
2023-02-06T09:04:12.081-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: @module=sdk.framework tf_attribute_path=AttributeName("id") tf_provider_addr=example.com/chrismarget/issue644 tf_req_id=38190b69-4c1a-a9bb-7420-8a503757b6ac @caller=/Users/bflad/src/github.com/hashicorp/terraform-plugin-framework/internal/fwserver/server_planresourcechange.go:364 tf_resource_type=issue644_rack_type tf_rpc=PlanResourceChange timestamp=2023-02-06T09:04:12.081-0500
2023-02-06T09:04:12.081-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: @module=sdk.framework tf_req_id=38190b69-4c1a-a9bb-7420-8a503757b6ac tf_attribute_path="AttributeName("leaf_switches").ElementKeyValue(tftypes.Object["logical_device":tftypes.Object["name":tftypes.String, "panels":tftypes.List[tftypes.Object["columns":tftypes.Number, "port_groups":tftypes.List[tftypes.Object["port_count":tftypes.Number, "port_roles":tftypes.Set[tftypes.String], "port_speed":tftypes.String]], "rows":tftypes.Number]]], "name":tftypes.String, "redundancy_protocol":tftypes.String, "spine_link_count":tftypes.Number, "spine_link_speed":tftypes.String]<"logical_device":tftypes.Object["name":tftypes.String, "panels":tftypes.List[tftypes.Object["columns":tftypes.Number, "port_groups":tftypes.List[tftypes.Object["port_count":tftypes.Number, "port_roles":tftypes.Set[tftypes.String], "port_speed":tftypes.String]], "rows":tftypes.Number]]]<null>, "name":tftypes.String<"leaf switch label">, "redundancy_protocol":tftypes.String<null>, "spine_link_count":tftypes.Number<"1">, "spine_link_speed":tftypes.String<"10G">>).AttributeName("logical_device")" tf_provider_addr=example.com/chrismarget/issue644 tf_resource_type=issue644_rack_type tf_rpc=PlanResourceChange @caller=/Users/bflad/src/github.com/hashicorp/terraform-plugin-framework/internal/fwserver/server_planresourcechange.go:364 timestamp=2023-02-06T09:04:12.081-0500

Inspecting the protocol level data to determine why the framework determined Detected value change between proposed new state and prior state is a little trickier for troubleshooting outside being able to run this in an acceptance test. That data is directly from Terraform. The TF_LOG_SDK_PROTO_DATA_DIR environment variable can be used to dump MessagePack encoded files, e.g. TF_LOG_SDK_PROTO_DATA_DIR=/tmp terraform plan -refresh=false and inspecting TIMESTAMP_PlanResourceChange_Request_PriorState.msgpack versus TIMESTAMP_PlanResourceChange_Request_ProposedNewState.msgpack.

Another option is starting the provider in debug mode, after adding code like this: Plugin Development - Debugging Framework Providers | Terraform | HashiCorp Developer

Which via editor configuration or manually via delve should provide a generated environment variable to trigger Terraform to use the debugger-attached, already running provider:

Provider started. To attach Terraform CLI, set the TF_REATTACH_PROVIDERS environment variable with the following:

        TF_REATTACH_PROVIDERS='{"example.com/chrismarget/issue644":{"Protocol":"grpc","ProtocolVersion":6,"Pid":75532,"Test":true,"Addr":{"Network":"unix","String":"/var/folders/f3/2mhr8hkx72z9dllv0ry81zm40000gq/T/plugin1988883645"}}}'

Then setting that generated environment variable value before executing Terraform. Using the debugger-guided output (since I was having trouble using fq on the MessagePack files directly), it appears that Terraform version 1.3.7 is sending a known value in the prior state for logical_devices while its sending a null value over in the proposed new state (plan) data:

This type of value difference will trigger the framework to mark any Computed and unconfigured attributes as unknown in the plan. Terraform core will use its typical set-based difference rules (e.g. the whole set value is the “index” for a set element) when rendering a plan for that type. Running off a main branch build of Terraform seems to yield the same result.

I think in terms of next steps here, there are two options:

  • One, potentially filing a Terraform core issue to see if its possible to associate the underlying Computed data correctly in the proposed new state from the available data in the prior state: Issues · hashicorp/terraform · GitHub
  • Two, trying to disassociate Computed attributes from the configurable set attribute. In general, sets which contain both configurable and computed data are problematic as by definition a set is “indexed” on its whole value. When there is partially configured data, the set “index” is different than its complete value. This can confuse logic on both sides of protocol.

Hi @bflad,

Thank you for this thorough reply. I especially appreciate your taking the time to walk me through your diagnostic workflow.

It’ll take me a while to fully process your reply, but restructuring the schema is definitely on the table as a possibility. You may remember I asked you about more sensibly handling sets of complicated objects during the partner briefing call on plugin framework shortly before the 1.x release about 6 weeks ago…

You said something along the lines of: “If your set object has an element which can be treated as a key, leverage it to convert the set to a map.”

This possibility has been on my mind since that day.

The username threw me off, but now I remember. :smile:

That is definitely another good general recommendation here – if you can use a map over a set, I would definitely suggest it. Maybe treating the leaf switch name as the map key can work without the data conversion being too difficult between your API and Terraform. Then hopefully that can get you unstuck here. Cheers!

I can convert the leaf_switches set to a map keyed by name.

In fact, the provider started out that way, but there are other elements (ethernet cables) in the rack_type object which need to reference switches by name. For reasons I can’t completely articulate/defend, setting up the schema so that attribute data (link target) winds up referencing attribute metadata (map key) felt wrong:

resource "issue644_rack_type" "my_rack" {
  name                       = "my rack"
  fabric_connectivity_design = "l3clos"
  leaf_switches              = {
    leaf_1 = {
#      name = "leaf_1" # this field is now the map key
      logical_device_id = "thing_with_48_ports"
      spine_link_count = 1
      spine_link_speed = "1G"
    }
  }
  leaf_switches              = {
    leaf_2 = {
#      name = "leaf_2" # this field is now the map key
      logical_device_id = "thing_with_48_ports"
      spine_link_count = 1
      spine_link_speed = "1G"
    }
  }
  servers = {
    server_1 = {
      logical_device_id = "thing_with_2_ports"
      links = {
        link_1 = {
          speed = "1G"
          target = "leaf_1" # <--- This string references a key to the `leaf_switches` map
        }
        link_2 = {
          speed = "1G"
          target = "leaf_2" # <--- This string references a key to the `leaf_switches` map
        }
      }
    }
  }
}

It turns out that while the names of leaf switch objects are guaranteed unique in the API I’m consuming, they’re allowed to contain spaces, punctuation, unicode characters, etc…

So using those values as a map key in the first place seems sketchy, and something about having a string buried elsewhere within the object (server->link->target, above) pointing to a key feels bad too.

But I can learn to live with it. An object naming constraint imposed by the provider which limits object names to HCL map-key-friendly characters isn’t the end of the world.

Should I convert the deeply nested port_roles (set of strings like “server” and “uplink”) to a list as well, or does this set not run afoul of the general “sets are hairy” guidance, or do you think that one will be okay?

If at all possible, it’s typically recommended to decompose Terraform configurations into separate resources where possible. This enables practitioners to iteratively manage their infrastructure as desired (e.g. separate modules, separate teams managing separate resources). Sometimes the API doesn’t support this more granular modification though so the below has to be taken with a grain of salt for your situation.

For example if your API can support separate calls:

resource "examplecloud_rack" "example" {
  name = "..."
}

resource "examplecloud_rack_leaf_switch" "example" {
  rack = examplecloud_rack.example.name
  name = "..."
  logical_device_id = "..."
}

resource "examplecloud_rack_server" "example" {
  rack = examplecloud_rack.example.name
  name = "..."
}

resource "examplecloud_rack_server_link" "example" {
  server = examplecloud_rack_server.example.name
  name = "..."
  speed = "..."
  target = examplecloud_rack_leaf_switch.example.name
}

Terraform should allow “complex” map keys via quotes or parenthesis: Types and Values - Configuration Language | Terraform | HashiCorp Developer

If you are stuck with a more “monolithic” API for managing this, then the general advice with sets is to ensure they are either all-Computed or all-configurable (Required/Optional) so Terraform on sides of the protocol has all information it needs in the configuration or prior state to create the proposed new state successfully.

This API does not support granular modification.

I tried to force my way into granular/individual resources with a much less complicated resource early on and have been regretting it: The Update() method on the nested resources winds up retrieving the parent and calculating diffs, with some locking to keep multiple modifications to the parent from stepping on each other.

It’s… not good. I have learned a lesson there, and will be revisiting those early resources to turn them into monoliths.

Terraform should allow “complex” map keys

Yep, I realized after posting that most of the problem there was my IDE and its HCL parser. Further, I don’t need to impose constraints: The terraform user will figure this out on their own. I just wish I wasn’t pushing them to write “ugly” HCL when their org’s naming conventions wind up calling for complex map keys.

the general advice with sets is to ensure they are either all-Computed or all-configurable

Got it. For simple types (strings, ints) it’s no problem. Things only have the potential to get ugly when we’re talking about sets of objects.

I’m hoping to resolve these problems with schema changes. If you think mentioning this experience to the terraform core folks (one of the approaches you mentioned) will be helpful/useful to somebody else, I’ll do that too.

Thanks again for engaging with me here. I really appreciate the help.

Hi @bflad,

I’ve converted the set-of-objects-with-computed-values to a map, and am having similar results.

For example, changing the spine_link_count here causes the (computed) panels list to get wiped out:

  # issue644_rack_type.a will be updated in-place
  ~ resource "issue644_rack_type" "a" {
        id                         = "bv8m-5qarhefnnhhrqqlfq"
      ~ leaf_switches              = {
          ~ "foo" = {
              ~ logical_device    = {
                    name   = "test"
                  ~ panels = [
                      - {
                          - columns     = 4 -> null
                          - port_groups = [
                              - {
                                  - port_count = 4 -> null
                                  - port_roles = [
                                      - "spine",
                                    ] -> null
                                  - port_speed = "10G" -> null
                                },
                            ]
                          - rows        = 2 -> null
                        },
                    ]
                }
                name              = "foo"
              ~ spine_link_count  = 3 -> 1
                # (2 unchanged attributes hidden)
            },
        }
        name                       = "aaa terraform"
        # (1 unchanged attribute hidden)
    }

I’ve updated the skeleton provider I linked previously to use the map-based layout, and to use yesterday’s commits to plugin-framework, and I’m definitely seeing the values get wiped out now:

2023-02-09T10:36:34.533-0500 [DEBUG] provider.terraform-provider-issue644: Detected value change between proposed new state and prior state: tf_provider_addr=example.com/chrismarget/issue644 tf_resource_type=issue644_rack_type @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:191 tf_attribute_path=leaf_switches @module=sdk.framework tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b tf_rpc=PlanResourceChange timestamp=2023-02-09T10:36:34.532-0500
2023-02-09T10:36:34.533-0500 [DEBUG] provider.terraform-provider-issue644: Marking Computed attributes with null configuration values as unknown (known after apply) in the plan to prevent potential Terraform errors: tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b tf_resource_type=issue644_rack_type @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:200 @module=sdk.framework tf_rpc=PlanResourceChange tf_provider_addr=example.com/chrismarget/issue644 timestamp=2023-02-09T10:36:34.533-0500
2023-02-09T10:36:34.533-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("id") tf_provider_addr=example.com/chrismarget/issue644 @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 @module=sdk.framework tf_resource_type=issue644_rack_type tf_rpc=PlanResourceChange tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b timestamp=2023-02-09T10:36:34.533-0500
2023-02-09T10:36:34.533-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("name") tf_rpc=PlanResourceChange @module=sdk.framework tf_provider_addr=example.com/chrismarget/issue644 tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b tf_resource_type=issue644_rack_type timestamp=2023-02-09T10:36:34.533-0500
2023-02-09T10:36:34.534-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 @module=sdk.framework tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b tf_rpc=PlanResourceChange tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("rows") tf_provider_addr=example.com/chrismarget/issue644 tf_resource_type=issue644_rack_type timestamp=2023-02-09T10:36:34.533-0500
2023-02-09T10:36:34.534-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("columns") tf_rpc=PlanResourceChange @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 @module=sdk.framework tf_resource_type=issue644_rack_type tf_provider_addr=example.com/chrismarget/issue644 tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b timestamp=2023-02-09T10:36:34.534-0500
2023-02-09T10:36:34.534-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: tf_rpc=PlanResourceChange tf_provider_addr=example.com/chrismarget/issue644 tf_resource_type=issue644_rack_type tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 @module=sdk.framework tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups").ElementKeyInt(0).AttributeName("port_roles") timestamp=2023-02-09T10:36:34.534-0500
2023-02-09T10:36:34.534-0500 [DEBUG] provider.terraform-provider-issue644: marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups").ElementKeyInt(0).AttributeName("port_speed") tf_provider_addr=example.com/chrismarget/issue644 tf_resource_type=issue644_rack_type tf_rpc=PlanResourceChange @module=sdk.framework tf_req_id=93b53c90-fca9-a07a-2d5a-ff1b891b746b @caller=/Users/cmarget/golang/pkg/mod/github.com/hashicorp/terraform-plugin-framework@v1.1.2-0.20230208224309-cfc0dc2c92a6/internal/fwserver/server_planresourcechange.go:364 timestamp=2023-02-09T10:36:34.534-0500

But I’m not sure what to do about it.

Is there still an issue with the way my schema is laid out?

edit: I may have found a clue while picking through the debug output.

Here’s a trimmed down section of the log output which mentions plan modifiers:

2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.String: tf_attribute_path=id
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.String: tf_attribute_path=id
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.String: tf_attribute_path=fabric_connectivity_design
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.String: tf_attribute_path=fabric_connectivity_design
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.String: tf_attribute_path=leaf_switches["foo"].name
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.String: tf_attribute_path=leaf_switches["foo"].name
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.Int64:  tf_attribute_path=leaf_switches["foo"].spine_link_count
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.Int64:  tf_attribute_path=leaf_switches["foo"].spine_link_count
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.Object: tf_attribute_path=leaf_switches["foo"].logical_device 
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.Object: tf_attribute_path=leaf_switches["foo"].logical_device 
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.String: tf_attribute_path=leaf_switches["foo"].logical_device.name
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.String: tf_attribute_path=leaf_switches["foo"].logical_device.name
2023-02-09T10:36:34.535-0500 [DEBUG] Calling provider defined planmodifier.List:   tf_attribute_path=leaf_switches["foo"].logical_device.panels
2023-02-09T10:36:34.535-0500 [DEBUG] Called  provider defined planmodifier.List:   tf_attribute_path=leaf_switches["foo"].logical_device.panels

The schema specifies plan modifiers for the following attribute paths:

leaf_switches["foo"].logical_device.panels[0].rows
leaf_switches["foo"].logical_device.panels[0].columns
leaf_switches["foo"].logical_device.panels[0].port_groups

…And those paths were “marked null” in the plan:

2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("rows")
2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("columns")
2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups")
2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups").ElementKeyInt(0).AttributeName("port_roles")
2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups").ElementKeyInt(0).AttributeName("port_speed")
2023-02-09T10:36:34.534-0500 [DEBUG] marking computed attribute that is null in the config as unknown: tf_attribute_path=AttributeName("leaf_switches").ElementKeyString("foo").AttributeName("logical_device").AttributeName("panels").ElementKeyInt(0).AttributeName("port_groups").ElementKeyInt(0).AttributeName("port_count")

But the plan modifiers associated with these attributes never ran.

Did these attributes vanish from the plan between the “mark as unknown” step and the invocation of the plan modifiers?

Perhaps marking the panels list “null” also left it empty?

I’ve reproduced the problem in a new provider with substantially simpler structure.

HERE

This one has two computed lists nested side-by-side within another attribute. One is a list of strings, the other is a list of objects.

The list of strings survives plan generation.

The list of objects does not survive plan generation.

@bflad, if you’re able to take another look, that would be super helpful.

1 Like

Hi again @hQnVyLRx and happy Friday to you.

Thank you very much for your continued troubleshooting and making it easier for anyone to reproduce the issue. This is a major time saver and it is very appreciated!

These reproductions helped discover a bug with nested attribute plan modification, which the fix has been submitted here: internal/fwserver: Use response plan value instead of request with nested attributes by bflad · Pull Request #669 · hashicorp/terraform-plugin-framework · GitHub

Using that updated framework code, the plan now shows as:

❯ terraform plan -refresh
╷
│ Warning: Provider development overrides are in effect
│
│ The following provider development overrides are set in the CLI configuration:
│  - example.com/chrismarget/copperfield in /Users/bflad/go/bin
│
│ The behavior may therefore not match any released version of the provider and applying changes may cause the state to become incompatible with published releases.
╵
copperfield_tour.sol_19830408: Refreshing state... [id=00f71e59-0df8-43da-8b32-8a1598365365]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # copperfield_tour.sol_19830408 will be updated in-place
  ~ resource "copperfield_tour" "sol_19830408" {
      ~ cities = {
          ~ "new_york" = {
              ~ season     = "spring" -> "summer"
                # (3 unchanged attributes hidden)
            },
        }
        id     = "00f71e59-0df8-43da-8b32-8a1598365365"
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Changes to Outputs:
  ~ x = {
      ~ cities = {
          ~ "new_york" = {
              ~ season     = "spring" -> "summer"
                # (3 unchanged elements hidden)
            }
        }
        id     = "00f71e59-0df8-43da-8b32-8a1598365365"
    }

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

One thing to potentially simplify the provider code in these cases now is that UseStateForUnknown() only needs to be applied to the “top level” Computed attributes, e.g.

						"venues": schema.ListNestedAttribute{
							Computed:      true,
							PlanModifiers: []planmodifier.List{listplanmodifier.UseStateForUnknown()},
							NestedObject: schema.NestedAttributeObject{
								Attributes: map[string]schema.Attribute{
									"capacity": schema.Int64Attribute{
										Computed: true,
									},
									"coordinates": schema.StringAttribute{
										Computed: true,
									},
								},
							},
						},

Thanks again for all your help here.

fix has been submitted here

Awesome!

I settled in with a sigh this morning: “Okay, today’s the day I learn how to run my provider in a debugger…” But you posted an update ~5 minutes earlier!

UseStateForUnknown() only needs to be applied to the “top level” Computed attributes

I’d been wondering about this, resorted to scattering them everywhere “just in case” :slight_smile:

I’m not completely clear where UseStateForUnknown() belongs when a computed+optional resource has nested elements within, but I’m confident that some experimentation will get me over this hump.

Thank you for your attention to this issue.