Bad ideas: JSON <===> tftypes.Value

I’m tinkering around with some bad idea code that invokes my framework provider’s resource and datasource CRUD methods directly. I’m not starting a provider server, and I don’t have (access to) model structs which can be used for conversion.

This plan(calling my CRUD methods) requires me to express the config, state, and plan as a tftypes.Value.

I’m starting with JSON data, so I was thrilled to discover the (deprecated) tftypes.ValueFromJSON() function. It’s exactly what I needed!

But I’ve not found an obvious way to run things in the opposite direction: turn a tftypes.Value into JSON-formatted []byte.

Before I go knock together a terrible recursive thing with knowledge of all simple, collection, and iterable in tftypes I figured I should ask…

Is there a right way to do this?

I don’t think there is an obvious way that you’re missing, unfortunately I think terraform-plugin-go/tftypes never added support for marshalling to JSON because it was just never needed (despite it being supported by Terraform’s type system :thinking:) .

The only usages of JSON for actual data in the provider protocol are in the Upgrade* RPCs, of which providers just need to unmarshal JSON state values from the request, then they send the data back as msgpack.


If you’re in the realm of “bad idea code” already :grinning_face: , you could consider following the flow of what happens in Terraform today and use go-cty for your JSON conversions, however it’s worth noting that the only Terraform data that can fully support that transition to JSON is state, since it’s guaranteed to always be fully known. Any unknown value can’t be converted to JSON, which is why most of the protocol uses msgpack :slightly_smiling_face:

Here’s a little toy example:

package provider

import (
	"fmt"
	"testing"

	"github.com/hashicorp/terraform-plugin-framework/resource"
	"github.com/hashicorp/terraform-plugin-go/tftypes"
	ctyjson "github.com/zclconf/go-cty/cty/json"
	ctymsgpack "github.com/zclconf/go-cty/cty/msgpack"
)

func TestRoundTrip(t *testing.T) {
	t.Parallel()

	// 1. Start with JSON data
	originalJSON := `{"attr_1":"hello","attr_2":"world!"}`

	// 2. Grab type from schema
	thingResource := NewThingResource() // <- returns a framework resource (resource.Resource) with two string attributes matching the JSON data
	schemaResp := resource.SchemaResponse{}
	thingResource.Schema(t.Context(), resource.SchemaRequest{}, &schemaResp)

	tfType := schemaResp.Schema.Type().TerraformType(t.Context())

	// 3. Convert JSON data to tftypes.Value
	tfValue, _ := tftypes.ValueFromJSON([]byte(originalJSON), tfType)
	fmt.Printf("tftypes.Value -> %s\n", tfValue)

	// 4. Convert tftypes.Value to msgpack
	msgpackData, _ := tfValue.MarshalMsgPack(tfType)

	// 5. Convert tftypes.Type to cty.Type (using JSON as a middle ground, which is how dynamic types work :D )
	tfTypeJSON, _ := tfType.MarshalJSON()
	ctyType, _ := ctyjson.UnmarshalType(tfTypeJSON)

	// 6. Convert msgpack to cty.Value
	ctyValue, _ := ctymsgpack.Unmarshal([]byte(msgpackData), ctyType)
	fmt.Printf("cty.Value -> %s\n", ctyValue.GoString())

	// 7. Convert cty.Value to JSON
	roundTrippedJSON, _ := ctyjson.Marshal(ctyValue, ctyType)

	// 8. Done!
	fmt.Printf("original JSON: %s\n", originalJSON)
	fmt.Printf("round tripped JSON: %s\n", roundTrippedJSON)
}

Should probably also include the relevant framework code and output of that test :laughing: , I also added an additional attribute just to show it’s being converted properly

func NewThingResource() resource.Resource {
	return &thingResource{}
}

type thingResource struct{}

func (r *thingResource) Schema(ctx context.Context, req resource.SchemaRequest, resp *resource.SchemaResponse) {
	resp.Schema = schema.Schema{
		Description: "Example resource.",
		Attributes: map[string]schema.Attribute{
			"attr_1": schema.StringAttribute{
				Required: true,
			},
			"attr_2": schema.StringAttribute{
				Computed: true,
			},
			"attr_3": schema.StringAttribute{
				Computed: true,
			},
		},
	}
}

Output from that test above:

tftypes.Value -> tftypes.Object["attr_1":tftypes.String, "attr_2":tftypes.String, "attr_3":tftypes.String]<"attr_1":tftypes.String<"hello">, "attr_2":tftypes.String<"world!">, "attr_3":tftypes.String<null>>

cty.Value -> cty.ObjectVal(map[string]cty.Value{"attr_1":cty.StringVal("hello"), "attr_2":cty.StringVal("world!"), "attr_3":cty.NullVal(cty.String)})

original JSON: {"attr_1":"hello","attr_2":"world!"}
round tripped JSON: {"attr_1":"hello","attr_2":"world!","attr_3":null}

Interesting. Thank you @austin.valle

I hadn’t clocked the “this only works for state” problem, but good news! That’s the data (the result of my CRUD operations) which I’m trying to get. So… Maybe no problem?

I wound up with this in my project’s internal/json package:

// ValueTo converts a tftypes.Value to JSON bytes, handling all Terraform types.
// This reverses what tftypes.ValueFromJSON does.
func ValueTo(val tftypes.Value) ([]byte, error) {
	// Convert the tftypes.Value to a generic Go value
	goVal, err := tftypesToGoValue(val)
	if err != nil {
		return nil, fmt.Errorf("converting tftypes.Value to go value: %w", err)
	}

	// Marshal to JSON
	data, err := json.Marshal(goVal)
	if err != nil {
		return nil, fmt.Errorf("marshaling to json: %w", err)
	}

	return data, nil
}

// tftypesToGoValue recursively converts a tftypes.Value to a Go value
// that can be JSON-marshaled.
func tftypesToGoValue(val tftypes.Value) (interface{}, error) {
	// Handle null values
	if val.IsNull() {
		return nil, nil
	}

	// Handle unknown values (not representable in JSON)
	if !val.IsKnown() {
		return nil, fmt.Errorf("cannot convert unknown value to JSON")
	}

	// Get the type to determine how to handle the value
	typ := val.Type()

	// Try to handle as primitives first by attempting conversions
	var s string
	if err := val.As(&s); err == nil {
		return s, nil
	}

	var n *big.Float
	if err := val.As(&n); err == nil {
		if n == nil {
			return nil, nil
		}
		f, _ := n.Float64()
		return f, nil
	}

	var b bool
	if err := val.As(&b); err == nil {
		return b, nil
	}

	// Handle collection types
	switch typ.(type) {
	case tftypes.List:
		return convertList(val)

	case tftypes.Set:
		return convertSet(val)

	case tftypes.Tuple:
		return convertTuple(val)

	case tftypes.Map:
		return convertMap(val)

	case tftypes.Object:
		return convertObject(val)

	default:
		return nil, fmt.Errorf("unsupported type: %T", typ)
	}
}

func convertList(val tftypes.Value) (interface{}, error) {
	var values []tftypes.Value
	if err := val.As(&values); err != nil {
		return nil, fmt.Errorf("converting list: %w", err)
	}

	result := make([]interface{}, len(values))
	for i, v := range values {
		goVal, err := tftypesToGoValue(v)
		if err != nil {
			return nil, fmt.Errorf("converting list element %d: %w", i, err)
		}
		result[i] = goVal
	}
	return result, nil
}

func convertSet(val tftypes.Value) (interface{}, error) {
	var values []tftypes.Value
	if err := val.As(&values); err != nil {
		return nil, fmt.Errorf("converting set: %w", err)
	}

	result := make([]interface{}, len(values))
	for i, v := range values {
		goVal, err := tftypesToGoValue(v)
		if err != nil {
			return nil, fmt.Errorf("converting set element %d: %w", i, err)
		}
		result[i] = goVal
	}
	return result, nil
}

func convertTuple(val tftypes.Value) (interface{}, error) {
	var values []tftypes.Value
	if err := val.As(&values); err != nil {
		return nil, fmt.Errorf("converting tuple: %w", err)
	}

	result := make([]interface{}, len(values))
	for i, v := range values {
		goVal, err := tftypesToGoValue(v)
		if err != nil {
			return nil, fmt.Errorf("converting tuple element %d: %w", i, err)
		}
		result[i] = goVal
	}
	return result, nil
}

func convertMap(val tftypes.Value) (interface{}, error) {
	var values map[string]tftypes.Value
	if err := val.As(&values); err != nil {
		return nil, fmt.Errorf("converting map: %w", err)
	}

	result := make(map[string]interface{}, len(values))
	for k, v := range values {
		goVal, err := tftypesToGoValue(v)
		if err != nil {
			return nil, fmt.Errorf("converting map value for key %q: %w", k, err)
		}
		result[k] = goVal
	}
	return result, nil
}

func convertObject(val tftypes.Value) (interface{}, error) {
	var values map[string]tftypes.Value
	if err := val.As(&values); err != nil {
		return nil, fmt.Errorf("converting object: %w", err)
	}

	result := make(map[string]interface{}, len(values))
	for k, v := range values {
		goVal, err := tftypesToGoValue(v)
		if err != nil {
			return nil, fmt.Errorf("converting object attribute %q: %w", k, err)
		}
		result[k] = goVal
	}
	return result, nil
}

Given the simple problem statement of “I want to fill config/plan/state as required and then run my CRUD methods”, I’m starting to wonder if I shouldn’t have just learned to use the provider plugin protocol rather than inventing my own nonsense.