Terraform-plugin-go dynamic pseudo type json marshal/unmarshalling

Hi,

I’m trying to add a dynamic pseudo type and i’m having difficulty un/marshalling unstructured data. My API client takes a map[string]interface{} so I have no structs I can use the ValueCreator, ValueConverter interfaces for.

The kuberneters alpha provider can easily marshal/unmarshal unstructured json objects into cty.Value but I haven’t seen anything to easily do this with plugin-go -

Can we add something like this to the terraform-plugin-go tftypes:

Your question is very timely! We’ve been discussing this exact problem.

Fundamentally, the issue at play here is that the tftypes.Value type can contain unknown values, and you’ll have no way of knowing if the values are unknown until runtime, because users can inject unknown values at any depth in any field. When converting a tftypes.Value to an interface{}, then, it’s unclear what should happen with unknown values.

I hope to have something to share on this front in the very near future, ideally this week. Technically it’s solvable outside the tftypes package, and I think that’s where I’d like to start with a solution, just because the tftypes package is very expensive for us to break compatibility on, so I want a high degree of confidence in things that go into it. I’m hoping to ship a small module with code in it that will let you do this, however, which solves for my concern because the module can have a major version bump without impacting the rest of the ecosystem.

Thanks @paddy, yeah i explored most of the new framework and then came to a grinding halt with this :smiley: look forward to seeing the new module - happy to help test out if needed.

Hey @iwarapter, sorry for the delay here. Had to get a new public repo spun up and ready for primetime, which takes a little doing. :slight_smile: You can find the package I’m talking about here. Disclaimers about being pre-release and not-heavily-used apply here, but it should be illustrative and usable.

Thanks @paddy will give it a go!

@paddy I have terraform-plugin-go-contrib working great for a dynamic input to my data source - thanks.
The implementation of the data source’s logic returns an interface{} which I’d now like to set in the value passed as the response’s State field.
Are you planning on adding a ValueCreator implementation to this package so that the interface{} can be converted to a Value, or is there some other way of doing this?

:+1: the GoPrimitive is working nicely. I’d love to see it implement the ValueCreator as well!

I hadn’t planned on implementing the ValueCreator interface, for perhaps non-intuitive reasons. When implementing the ValueConverter interface, we know what the input values are going to be: they’re one of our internal representations for one of the Terraform types, which is a limited set. Because of that, the job of the package is straightforward: pick a canonical version of each of those types and set GoPrimitive.Value to them.

ValueCreator has a more difficult proposition, because the range of input values we can handle is theoretically infinite–GoPrimitive.Value can be set to any Go type, and the expectation is that we’ll turn it into a tftypes.Value. This is, obviously, not something we’ll ever be able to satisfy fully.

But maybe there’s a middle ground here where we can create a tftypes.Value from a certain subset of types, say the built-in Go types and some chosen standard library primitives, and throw an error on everything else. That may strike the right balance of pragmatism, though it seems like a shaky premise to build a provider on to me. I’d love an enhancement issue for this, and we can work through the nuance there in a way that will get tied to the code in context for future maintainers wondering why things are the way they are.

Hi @paddy I defiantly think this is a feature that is needed!

I had a quick play at implementing this - I have been able to get something that works (it converts all the test cases for GoPrimitive back to their tftypes.Value.

However for this to work I needed a small patch of the plugin-go module to add a Type() method to return the Value type (i just vendored and added the project to demo it) - terraform-plugin-go-contrib/value.go at feature/add-value-creator-for-go-primitives · iwarapter/terraform-plugin-go-contrib · GitHub

However it’s doesn’t properly work with NewValue as it actually returns the ready to go tftypes.Value but NewValue then tries to type check it against the actual go primitives -

if creator, ok := val.(ValueCreator); ok {
	var err error
	val, err = creator.ToTerraform5Value()
	if err != nil {
		panic("error creating tftypes.Value: " + err.Error())
	}
}

This only works because I’m storing the type data when the value is originally unmarshalled and assumes its not changing.

Quick Update: I have tested this with one of my providers and its working as expected (for some basic dynamic types - will try find some more challenging scenarios

Anyways food for thought.