Storing data property in a different encoding (gzipped base64)

Terraform Providers Plugin Development
1

Hello!

I’m using Terraform Plugin Framework to develop a plugin that allows describing Grafana dashboards using HCL.

For example:

data "gdashboard_stat" "status" {
  title       = "Status"
  description = "Shows the status of the container"

  field {
    mappings {
      value {
        value        = "1"
        display_text = "UP"
        color        = "green"
      }

      special {
        match        = "null+nan"
        display_text = "DOWN"
        color        = "red"
      }
    }
  }

  queries {
    prometheus {
      uid     = "prometheus"
      expr    = "up{container_name='container'}"
      instant = true
    }
  }
}

# data.gdashboard_stat.status.json - returns the JSON representation of the panel

The json property is computed.

Unfortunately, the JSON output may become rather big when managing dashboards with many panels. As a result, the state file will grow, too. For example, 20+ dashboards with 20-30 panels each can easily occupy 20 MBs of the state file.

If the state file is stored in remote storage (e.g. S3), it takes 30-90 seconds only to download the file. Which makes operations notably slow.

To keep the state file compact, I would like to keep the JSON representation gzipped and base64 encoded.

Since base64gunzip does not exist in the terraform (Feature request: gunzip function · Issue #22568 · hashicorp/terraform · GitHub), I need to decode a value on-fly within the plugin.

Is it possible to do the following:

  1. Store JSON gzipped + base64 encoded in the state file
  2. While retrieving a value from the state - decode and gunzip to operate the normal JSON

If I get it right, I need some kind of a StateFunc from the old SDK. Alternatively, I was looking at custom types too. But from what I see, it does not provide what I need.

2

Alternatively, can a computed field be excluded from being written to the state file? When the data object is retrieved from the state, the value can be recomputed again.

3

Hey there @iRevive :wave: ,

… can a computed field be excluded from being written to the state file? When the data object is retrieved from the state, the value can be recomputed again.

It’s not currently possible to prevent Terraform from storing computed values in state, so in your case I’d say storing the compressed value is the way to go.

Is it possible to do the following:
1. Store JSON gzipped + base64 encoded in the state file
2. While retrieving a value from the state - decode and gunzip to operate the normal JSON

So as of Terraform 1.7, we don’t have a built-in function available that can perform #2, so the only options involve putting something in state.

Terraform 1.8 will introduce support for provider-defined functions, which you should be able to build a function that accepts a gzipped/base64 encoded value and then output whatever value you’d like:

locals {
  uncompressed_json = provider::gdashboard::base64gunzip(data.gdashboard_stat.status.gzipped_json)
}

data "gdashboard_stat" "status" {
   # computed "gzipped_json" attribute
}

Plugin Framework v1.5.0 has technical preview support for provider-defined functions functions and an alpha of Terraform 1.8 is available, which I believe has that support (no compatibility guarantees until released)

4

Just to be clear because I didn’t see it in the question, but is the json output useful for any other resources? If the data is not going to be referenced but you still need it within the provider for planing operations, then you can store it however you want because decoding is done only within your provider code.

If the data is not used for anything else within Terraform and only being stored because it is part of the API response, then you have the option to not store it at all.

5

Yes, the data is used by the grafana provider.

# Create your dashboard
resource "grafana_dashboard" "my_dashboard" {
  config_json = data.gdashboard_dashboard.dashboard.json
}
6

Thanks, I will take a look!

7

Just to make sure we’re on the same page, if you end up passing the uncompressed JSON to another resource attribute it will get saved to state there.

Example:

{
  "version": 4,
  "terraform_version": "1.8.0",
  "serial": 3,
  "lineage": "931b7fb2-7709-efdd-c169-09cdc5abd7e9",
  "outputs": {},
  "resources": [
    {
      "mode": "data",
      "type": "gdashboard_dashboard",
      "name": "dashboard",
      "provider": "provider[\"registry.terraform.io/gdashboard/gdashboard\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            // Compressed
            "gzipped_json": ""
          },
          "sensitive_attributes": []
        }
      ]
    }
    {
      "mode": "managed",
      "type": "grafana_dashboard",
      "name": "my_dashboard",
      "provider": "provider[\"registry.terraform.io/grafana/grafana\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            // Uncompressed
            "config_json": ""
          },
          "sensitive_attributes": []
        }
      ]
    }
  ],
  "check_results": null
}
8

That is an important thing to consider too when dealing with large amounts of data :point_up:, anything that uses this json blob needs to store it again in it’s own state.

Another option some providers use, is if this is retrievable via the API, and used by other services within that same API, you could lookup the data on demand via a unique identifier.

9

Passing an uncompressed resource to the grafana_dashboard is fine, because the grafana provider can be configured to store hashes rather than raw JSON (Terraform Registry).

However, the JSON property of the gdashboard_dashboard is stored in the state:

 {
      "module": "module.dashboard_jvm",
      "mode": "data",
      "type": "gdashboard_dashboard",
      "name": "dashboard",
      "provider": "provider[\"registry.terraform.io/gdashboard/gdashboard\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "annotations": [
              {
                "grafana": [],
                "prometheus": [ ]
              }
            ],
            "description": null,
            "editable": false,
            "graph_tooltip": "shared-crosshair",
            "id": "xxxxx",
            "json": "{\"uid\":\"production-jvm\", ....... }\" <- the big JSON is stored here
         }
      ...
      ]
}

And I would like to minimize the size of the gdashboard_dashboard in the state.

10

From what I see, an upcoming provider-defined function in Terraform 1.8 will solve the problem.

11

Ah, thanks for sharing that info. I took a look through the grafana code base and the store_dashboard_sha256 property is not following Terraform’s data consistency rules (storing a value in state that differs from the exact configuration value or prior state value).

This is specifically permitted (by Terraform) only when using the old SDKv2, but is not permitted with Plugin Framework. If you don’t maintain the grafana provider then it’s not your problem to solve, but eventually whenever Grafana goes to upgrade that specific resource from using SDKv2 to Plugin Framework, that functionality won’t be possible.

We wrote up some documentation about this data consistency problem in SDKv2 here: Resources - Data Consistency Errors | Terraform | HashiCorp Developer