Unfortunately I’m not sure I really follow exactly what you are trying to achieve here, and so it’s hard to give specific advice.
With that said, in general
go-plugin with gRPC can do anything gRPC can do, which is constrained mainly by the Protocol Buffers encoding format and associated schema language.
Protocol Buffers has a few different options to deal with structures that can vary dynamically:
OneOf lets you create a tagged union field, which for Go is like an interface with a closed set of implementations defined in the same package. This allows different calls to use different types so long as you can enumerate all of the possible types statically in the schema.
Any is an even more general option where the message on the wire includes both the data and an identifier indicating the type of the data. Because Protocol Buffers data is not self-describing, the recipient of the message will still need to have some way to find a schema for the indicated data type, but you can support new types in the future without changing the main protocol schema, as long as both the plugin and its caller agree on what data types are valid. I’ve not tried this with Go in particular, but I assume it would end up making a Go struct field that takes any implementation of the protobuf
- Stepping outside of what is representable directly in Protocol Buffers, you can declare a field of type
bytes at the protocol buffers level and then in your application code write in data serialized in some other self-describing format, like JSON or msgpack. This similar to using
Any but by using a self-describing format you can potentially avoid the need for both sides to agree ahead of time on a schema, if the task at hand allows working generically with a data structure.
Terraform uses the third of these options because it has its own higher-level sense of schema from the Terraform language itself, and so the wire format just passes around msgpack or JSON-serialized data which Terraform Core and the SDK can then parse and decode using the Terraform-language-oriented schema.
If you can share a little more about what you’ve tried and what happened when you tested it then I might be able to give some more specific pointers. However, when I’m working on API design for gRPC-based things I honestly usually just look for other APIs that use gRPC that have solved similar problems and see how well their solutions apply to my problem.
The other tactic I generally follow is to minimize the direct use of the types generated from the Protocol Buffers schema, which are mechanically-generated and therefore inevitably non-idiomatic for the specific language I’m working in, and instead I wrap them in a custom-designed API that is more appropriate for the target language. The Terraform team followed this principle when implementing the Terraform plugin protocol version 5, whereby the auto-generated package is wrapped by a hand-written interface that the rest of the application consumes. In that way, the awkward Go types that result from some of the Protocol Buffers constructs are hidden, and the rest of the application can see a more convenient and idiomatic API.