Go-plugin interface design

Hi! I’m looking to use the go-plugin framework with gRPC and I was wondering if there were any best practices for interface design.

I’d like to have a common interface with a handful of methods that all plugins could call, but each plugin could have different arguments to those methods.

I normally would get around this by instantiating a struct first, but it doesn’t seem like I can do this with the go-plugin framework.

Would I be forced to use the empty interface in go for all method signatures?

Thanks!

Hi @bcatubig,

Unfortunately I’m not sure I really follow exactly what you are trying to achieve here, and so it’s hard to give specific advice.

With that said, in general go-plugin with gRPC can do anything gRPC can do, which is constrained mainly by the Protocol Buffers encoding format and associated schema language.

Protocol Buffers has a few different options to deal with structures that can vary dynamically:

  1. OneOf lets you create a tagged union field, which for Go is like an interface with a closed set of implementations defined in the same package. This allows different calls to use different types so long as you can enumerate all of the possible types statically in the schema.
  2. Any is an even more general option where the message on the wire includes both the data and an identifier indicating the type of the data. Because Protocol Buffers data is not self-describing, the recipient of the message will still need to have some way to find a schema for the indicated data type, but you can support new types in the future without changing the main protocol schema, as long as both the plugin and its caller agree on what data types are valid. I’ve not tried this with Go in particular, but I assume it would end up making a Go struct field that takes any implementation of the protobuf Message interface.
  3. Stepping outside of what is representable directly in Protocol Buffers, you can declare a field of type bytes at the protocol buffers level and then in your application code write in data serialized in some other self-describing format, like JSON or msgpack. This similar to using Any but by using a self-describing format you can potentially avoid the need for both sides to agree ahead of time on a schema, if the task at hand allows working generically with a data structure.

Terraform uses the third of these options because it has its own higher-level sense of schema from the Terraform language itself, and so the wire format just passes around msgpack or JSON-serialized data which Terraform Core and the SDK can then parse and decode using the Terraform-language-oriented schema.

If you can share a little more about what you’ve tried and what happened when you tested it then I might be able to give some more specific pointers. However, when I’m working on API design for gRPC-based things I honestly usually just look for other APIs that use gRPC that have solved similar problems and see how well their solutions apply to my problem.

The other tactic I generally follow is to minimize the direct use of the types generated from the Protocol Buffers schema, which are mechanically-generated and therefore inevitably non-idiomatic for the specific language I’m working in, and instead I wrap them in a custom-designed API that is more appropriate for the target language. The Terraform team followed this principle when implementing the Terraform plugin protocol version 5, whereby the auto-generated package is wrapped by a hand-written interface that the rest of the application consumes. In that way, the awkward Go types that result from some of the Protocol Buffers constructs are hidden, and the rest of the application can see a more convenient and idiomatic API.