Provider frameworks for {Python, TypeScript, anything-non-Go}

What’s the current attitude to plugins/providers built in programming languages other than Go, and is there any hope of adding official frameworks/SDKs for these in future?

I’m relatively new to TF, but have a lot of experience with AWS CloudFormation & CDK. I’ve often had to write CloudFormation custom resources in the past to bridge (usually temporary) gaps in CFn coverage… And although that system has its fair share of problems it does enable us to write custom resources in a range of high-level languages - which I really appreciate since most of my work is usually in Python & TypeScript. Picking up Go from scratch seems like a heavy lift just to hack together IaC integration for a new CRUDL API.

As I understood so far (from exploring the official terraform-plugin-framework, the third-party Python-TF plugin framework):

  • Under the hood, a provider/plugin is a standalone process implementing a gRPC server, but the TF docs emphasise the plugin framework (based on terraform-plugin-go) as the point of integration rather than that low-level RPC interface. I think the RPCs are documented here?
  • Terraform bakes in an assumption that a plugin is a standalone executable binary, which Go makes significantly easier than Python & JavaScript - but there are packaging options out there for both…
  • Python-TF caveats makes some comments about assumed reluctance of HashiCorp to entertain partnering with plugins that are built on more custom frameworks for the registry.

From the fact I couldn’t find loads of existing threads here or GitHub issues asking for additional language bindings already, I assume the demand’s not been evidenced yet to commit to supporting more SDKs. The most realistic route seems like it’d be to build adoption of providers based on 3p-maintained frameworks first, to prove interest?

So I’m wondering if we could get clarity on whether HashiCorp really would resist or block a plugin/provider from partnering with the registry if it was based on a direct RPC integration rather than the proper SDK/framework (but was still able to provide standalone executable files as expected)?

Hey there @ath :waving_hand: , I’m one of the maintainers working on Terraform’s protocol and plugin SDKs.

We have a section in our best practices docs about this that you might have seen, which still holds true: Provider code | Terraform | HashiCorp Developer .

I’ll try to share some more for those interested :eyes:


So I’m wondering if we could get clarity on whether HashiCorp really would resist or block a plugin/provider from partnering with the registry if it was based on a direct RPC integration rather than the proper SDK/framework (but was still able to provide standalone executable files as expected)?

In general, if you open a GitHub issue against Terraform core that’s related to a provider, we will assume it’s written in Go and served via terraform-plugin-go, using it’s type system tftypes , etc. If it’s not, there’s only so much effort we can go through to determine what problems you’re running into.

Under the hood, a provider/plugin is a standalone process implementing a gRPC server, but the TF docs emphasise the plugin framework (based on terraform-plugin-go) as the point of integration rather than that low-level RPC interface. I think the RPCs are documented here?

Since the protocol has been changing so much recently, I’d say the best documentation of it is the source itself. Typically the server-side of the protocol is more documented since we export these packages:

There are a lot of “undocumented” parts of the protocol (i.e. assumptions about the client or server), for example:

  • Which RPCs are guaranteed to be called by certain commands and in what order
  • What data is valid to be returned by an RPC

The most complicated RPCs are the ones involved with Terraform plan/apply, which do have some documentation that touches on some of those data constraints: terraform/docs/resource-instance-change-lifecycle.md at main · hashicorp/terraform · GitHub

Other RPCs like the ones involved for provider-defined functions are much more straightforward.

The main assumption the protocol makes is the encoding/decoding of data (DynamicValue in the protocol), which is all based off the zclconf/go-cty Go module that Terraform core/HCL uses:

Terraform bakes in an assumption that a plugin is a standalone executable binary, which Go makes significantly easier than Python & JavaScript - but there are packaging options out there for both…

Right, specifically it needs to be a binary that communicates the same as `hashicorp/go-plugin`, which Terraform uses as a client and the providers use to serve. So anyone implementing a provider framework would want to reference that project: GitHub - hashicorp/go-plugin: Golang plugin system over RPC.


Hopefully some of that information helps :slightly_smiling_face: