Terraform 0.12 -> 0.13 upgrade failed with "cannot decode dynamic from flatmap" (remote data source)

Terraform
1

Background:

Terraform 0.12.31 project with 2 remote data sources (both data sources states are 0.12.31, prev upgraded from 0.11.15)

During project upgrade 0.12.31 → 0.13.7 faced issue:

Error: cannot decode dynamic from flatmap

trace says that it’s an issue with remote data sources

2022/01/25 09:53:28 [TRACE] [walkRefresh] Entering eval tree: data.terraform_remote_state.level2_sd-wan_network
2022/01/25 09:53:28 [TRACE] eval: *terraform.EvalSequence
2022/01/25 09:53:28 [TRACE] eval: *terraform.EvalGetProvider
2022/01/25 09:53:28 [TRACE] eval: *terraform.EvalReadState
2022/01/25 09:53:28 [TRACE] EvalReadState: reading state for data.terraform_remote_state.level2_sd-wan_network
2022/01/25 09:53:28 [ERROR] eval: *terraform.EvalReadState, err: cannot decode dynamic from flatmap
2022/01/25 09:53:28 [ERROR] eval: *terraform.EvalSequence, err: cannot decode dynamic from flatmap
2022/01/25 09:53:28 [TRACE] [walkRefresh] Exiting eval tree: data.terraform_remote_state.level2_sd-wan_network
2022/01/25 09:53:28 [TRACE] vertex "data.terraform_remote_state.level2_sd-wan_network": visit complete
2022/01/25 09:53:28 [TRACE] vertex "data.terraform_remote_state.level2_sd-wan_network": dynamic subgraph encountered errors
2022/01/25 09:53:28 [TRACE] vertex "data.terraform_remote_state.level2_sd-wan_network": visit complete
2022/01/25 09:53:28 [TRACE] vertex "data.terraform_remote_state.level2_sd-wan_network (expand)": dynamic subgraph encountered errors
2022/01/25 09:53:28 [TRACE] vertex "data.terraform_remote_state.level2_sd-wan_network (expand)": visit complete

I am aware of such a problem when 0.11.x reads 0.12/0.13 remote data sources but I already upgraded remote data sources to 0.12

{
  "version": 4,
  "terraform_version": "0.12.31",
  "serial": 19,
  "lineage": "4db5d032-94b4-95bd-5c20-e24900db8840",
  "outputs": {
    "az1_storage_nsgflow_id": {
      "value": "XXX",
      "type": "string"
    }

also, overall my remote data sources look pretty straightforward → strings.

Any ideas that can be the problem with? Something specific that I can check in my remote data sources (outputs), format? etc?

2

Hi @hayorov,

I’ve only seen mention of this once before, but unfortunately never received any follow up from that user.

I don’t think there was any way to execute an old terraform provider binary with v0.13, because that provider was built-int, but you could check the early init trace logs to make sure you are not somehow picking up an old plugin.

I think the most likely situation is that there is a state file which has not been fully upgraded, and contains a legacy representation of the remote state data. You can check the state files manually for attributes_flat objects under any remote state instances, which would explain the error given. Hopefully this is just a matter of an oversight, and refreshing the problematic state with the correct version will remedy the situation.

1 Like
3

Many thx, @jbardin I’ll check that tomorrow morning.

May I double-check that ‘terraform refresh’ is the right way to update the remote state once the codebase is updated with -0.XXupgrade?

4

I have a feeling that in this case you may need to manually apply, even if there are no planned changes, or even skip refresh entirely. The error is happening during refresh in this case, so if you are sure that the rest of the resources are generally in sync, perhaps skipping the refresh will allow the data source to be re-written correctly. The separate “refresh” phase was removed in later releases, which also fixed many issues with data sources.

5

It sounds like we’re talking about a legacy state entry for a data resource instance here – an instance of terraform_remote_state – in which case an alternative path forward might be to ask Terraform to “forget” the previous values and thus treat this as if you’d just added that data block for the first time:

terraform state rm data.terraform_remote_state.level2_sd-wan_network

“Forgetting” a resource in this way requires some care when dealing with managed resources (those declared with resource blocks) because Terraform uses the state to remember which object it’s managing, but data resources (with data blocks) are just references to objects managed elsewhere and so just the configuration should be enough to fetch the same data, without any help from the prior state.

FWIW, this is something that ought to have been dealt with by running terraform apply while on Terraform v0.12, which had some extra logic to help upgrade over this bump. It seems like that didn’t work out for you for some reason, but it’ll be hard to debug that in retrospect and so I think it’d be better to just work around it and move forward, using one of the approaches discussed here.

6

Many thanks, @jbardin, and @apparentlymart,

Indeed, project state file with:

{"version": 4, "terraform_version:" "0.13.7"}

has

{
    "mode": "data",
    "type": "terraform_remote_state",
    "name": "level2_sd-wan_network",
    "provider": "provider[\"terraform.io/builtin/terraform\"]",
    "instances": [
      {
        "schema_version": 0,
        "attributes_flat": {
          "az1_gcp_vnet_id": "",
          "az1_gcp_vnet_name": "",
          "az1_gcp_vnet_rg": "",
          "az1_lbu_vnet_id.#": "2",
...
}

as @jbardin suggested to inspect with keyword attributes_flat.

Sum up: The W/A by @apparentlymart with remote data source state rm worked.

It seems that terraform refresh cannot be used as a reliable strategy to upgrade 0.11->0.12->0.13->… and apply is strongly preferred. Unfortunately for some reasons apply cannot be blindly (massively) used in our case for an upgrade. Overall refresh side-effects could be w/a.

PS. It seems that one more refresh side-effect could be a module’s provider format ("/-/") partial update.

we can close the topic. Great support :slight_smile: