SDK/Provider Development: Anyone ever used code generation or other tools to simplify their provider development?

Hi Everyone!

I am curious if anyone has ever utilized code generation or other tools to automate some of their Terraform provider development? I ask because when developing a provider it feels a bit like I am acting as a translator from an API to a terraform resource, but if the API is defined in a robust way then perhaps a provider developer acting as translator is no longer needed.

Some examples of this could be things like:

Please share your links and thoughts.

The hashicorp/google provider and the hashicorp/google-beta provider are both generated in large part from the Google Cloud Platform “magic modules” codebase. This is the most thorough effort I’ve seen to generate Terraform provider code, where magic modules uses a Ruby-based DSL to define the different object types which can then get translated into other forms, including Terraform provider code.

Both of the providers you mentioned are more dynamic, aiming to bind to a particular API at runtime rather than development time. Terraform’s plugin model isn’t really designed to support that development model and so they both have some unusual usage patterns to work around that. They can certainly be useful for one-off integrations with in-house APIs, and similar situations where maintaining a full provider codebase seems undesirable, but I think the Google Cloud provider approach of generating code at development time, and shipping a binary with all of the necessary stuff built in, will be the most ergonomic from an end-user standpoint.

Terraform v0.13 and later for the first time allow providers to include other files in their packages alongside the plugin executable, which they can then access at runtime, so although I’m not aware of anyone having tried this in practice yet I think there’s an theoretical compromise of building a provider a bit like the ones you shared here but which looks for the OpenAPI/etc definition relative to os.Args[0] (the executable path), rather than via environment variables, and then you could theoretically bundle that executable up with an OpenAPI schema together in a .zip file and have a provider that behaves just like any other as far as end-users are concerned, even though its implementation is actually dynamic based on the contents of the schema:

  • terraform-provider-example - a copy of a generalized provider executable that looks for a schema in its directory and derives its behavior from that.
  • example-schema.json - the filename where terraform-provider-example expects to find the schema it’ll use.

This would then, in principle, allow just updating the schema file in future releases and leaving the executable entirely unchanged, unless you need to add support for additional OpenAPI features that the initial implementation didn’t include.

Thanks for the reply!

The magic modules is pretty much what I was looking for. Though they took a step further than what I had in mind and build for multiple targets. It does look a bit more complex than I imagined, I wonder if doing something like magic modules only makes sense if your API surface is massive. Or perhaps there could be a tool for creating a Terraform provider DSL to reduce the barrier to entry for smaller orgs.

The notion of a semi-dynamic provider like you described seems like the ideal middle ground. I think the key is really having a machine-readable source of truth for your API. With the goal of having product developers be responsible for defining the exact behavior of their API.