SDK/Provider Development: Anyone ever used code generation or other tools to simplify their provider development?

Terraform Providers Plugin Development
1

Hi Everyone!

I am curious if anyone has ever utilized code generation or other tools to automate some of their Terraform provider development? I ask because when developing a provider it feels a bit like I am acting as a translator from an API to a terraform resource, but if the API is defined in a robust way then perhaps a provider developer acting as translator is no longer needed.

Some examples of this could be things like:

Please share your links and thoughts.

1 Like
2

The hashicorp/google provider and the hashicorp/google-beta provider are both generated in large part from the Google Cloud Platform “magic modules” codebase. This is the most thorough effort I’ve seen to generate Terraform provider code, where magic modules uses a Ruby-based DSL to define the different object types which can then get translated into other forms, including Terraform provider code.

Both of the providers you mentioned are more dynamic, aiming to bind to a particular API at runtime rather than development time. Terraform’s plugin model isn’t really designed to support that development model and so they both have some unusual usage patterns to work around that. They can certainly be useful for one-off integrations with in-house APIs, and similar situations where maintaining a full provider codebase seems undesirable, but I think the Google Cloud provider approach of generating code at development time, and shipping a binary with all of the necessary stuff built in, will be the most ergonomic from an end-user standpoint.

Terraform v0.13 and later for the first time allow providers to include other files in their packages alongside the plugin executable, which they can then access at runtime, so although I’m not aware of anyone having tried this in practice yet I think there’s an theoretical compromise of building a provider a bit like the ones you shared here but which looks for the OpenAPI/etc definition relative to os.Args[0] (the executable path), rather than via environment variables, and then you could theoretically bundle that executable up with an OpenAPI schema together in a .zip file and have a provider that behaves just like any other as far as end-users are concerned, even though its implementation is actually dynamic based on the contents of the schema:

  • terraform-provider-example - a copy of a generalized provider executable that looks for a schema in its directory and derives its behavior from that.
  • example-schema.json - the filename where terraform-provider-example expects to find the schema it’ll use.

This would then, in principle, allow just updating the schema file in future releases and leaving the executable entirely unchanged, unless you need to add support for additional OpenAPI features that the initial implementation didn’t include.

1 Like
3

Thanks for the reply!

The magic modules is pretty much what I was looking for. Though they took a step further than what I had in mind and build for multiple targets. It does look a bit more complex than I imagined, I wonder if doing something like magic modules only makes sense if your API surface is massive. Or perhaps there could be a tool for creating a Terraform provider DSL to reduce the barrier to entry for smaller orgs.

The notion of a semi-dynamic provider like you described seems like the ideal middle ground. I think the key is really having a machine-readable source of truth for your API. With the goal of having product developers be responsible for defining the exact behavior of their API.

4

Just to add another data-point ; we put together a terraform provider code generator tool. It inputs from an OpenAPI spec, and outputs a terraform provider compliant with the terraform-plugin-framework. In theory it’s indistinguishable from a manually written one. We did a bit of a mix between the magic modules approach and terraform-provider-openapi: using code generation to “compile” a terraform-plugin-framework compliant provider from an OpenAPI spec.

I’m biased, but I think it works really nicely, because most people have OpenAPI specification generators configured to use their server routes code / types as the source of truth, which means API changes “flow” into terraform providers without any manual effort.

  1. Whenever server code changes, the OpenAPI specification changes through tooling like zod-to-openapi (there’s something for most server frameworks).
  2. That OpenAPI specification is hosted automatically as part of deploying new server code.
  3. We configure pollers that watch the hosting URL of the specification, and when they spot a change:
  4. The terraform provider is regenerated, triggering a PR against the terraform provider repository.
  5. Which has tests run automatically against it, and when merged.
  6. Automatically releases to the terraform registry.

[0] https://github.com/speakeasy-api/speakeasy
[1] Toy Example https://github.com/speakeasy-sdks/terraform-provider-hashicups

4 Likes
6

Hi all :wave:,

The Terraform DevEx team has recently released a tech preview that’s aiming to put some guardrails around the provider code generation problem space. This isn’t intended to be an “all-in-one” code generator, but rather general pieces that can fit together to build a code generation solution that works for specific providers.

Here, the problem is broken into 3 pieces, with an implementation to showcase potential solutions for each. Developers may be able to get value out of all the solutions, or maybe just a few of them, but hopefully it will reduce the amount of custom code generation a maintainer needs to write.

The blog post and documentation link above contain a lot of this info, but I’ll summarize the problem and solutions here as well. Here is a high-level summary of the design:

Code Gen Solution Diagram
Code Gen Solution Diagram893Ă—273 22.8 KB

1. Representing a Terraform provider

In order to break this problem into pieces, we first need an intermediate representation of a Terraform provider to separate some concerns. We created the provider code specification to serve this purpose, which is just a JSON schema that defines how a Terraform provider is modeled.

If you’re already familiar with provider development you’ll notice it’s mostly schema information at the moment, with some custom Go bits like imports.

The full JSON schema definition is here.

2. Translating from source(s) to a Terraform provider

A “source” most commonly refers to an IDL like an OpenAPI specification or Protobuf, but a source doesn’t necessarily need to be an IDL. There also may be more than one source, could be multiple OpenAPI specs, a protobuf file and an API SDK, etc.

A provider spec generator takes in one or more sources and generates a provider code specification. This process needs to translate how a source, like an OpenAPI specification, maps to Terraform’s concepts such as data handling. This problem ranges in complexity depending on what source(s) your provider’s API uses.

For a source like Cloud Formation Resource Schema, the semantics of resources map pretty well to Terraform’s data handling concepts, making the translation process easier. The awscc provider is generated based on this source today.

For a more generic source like OpenAPI, there is more complexity/customization in determining the proper mapping to a Terraform provider. We built the OpenAPI provider spec generator that showcases one potential solution for translating this kind of source. This solution relies on a configuration file to drive that customization.

3. Generating Terraform provider stuff

Once you have a provider code specification, you can start generating the Terraform provider. This most likely will be generating actual Terraform provider Go code, but isn’t necessarily limited to just this. You could also use a provider code specification to generate tests, documentation, etc.

We built the Plugin Framework code generator that generates the Terraform SDK Go code for managed resources, data source, or a provider. As the name suggests, it generates Go code that uses the Plugin Framework SDK and currently it focuses on generating schema code.

Putting the pieces together

We wrote a workflow example that shows a sample workflow for how these tools could be used together. Currently it’s CLI focused, but there are no technical restrictions that prevent more CI friendly approaches.

1 Like
8

Hi @austin.valle,

Thank you so much for this! I’ve been developing a provider and have been hoping for a tool like this one.
I went through and followed the workflow example but when I get to the point where I have to go install . to build the bin with the newly generated pet_resource_gen.go code I get a ton of duplicate functions from the openapi generation:

go install .
# terraform-provider-petstore/internal/resource_pet
internal/resource_pet/pet_resource_gen.go:1908:6: NameType redeclared in this block
        internal/resource_pet/pet_resource_gen.go:438:6: other declaration of NameType
internal/resource_pet/pet_resource_gen.go:1978:6: NewNameValueNull redeclared in this block
        internal/resource_pet/pet_resource_gen.go:584:6: other declaration of NewNameValueNull
internal/resource_pet/pet_resource_gen.go:1984:6: NewNameValueUnknown redeclared in this block
        internal/resource_pet/pet_resource_gen.go:590:6: other declaration of NewNameValueUnknown
internal/resource_pet/pet_resource_gen.go:1990:6: NewNameValue redeclared in this block
        internal/resource_pet/pet_resource_gen.go:596:6: other declaration of NewNameValue
internal/resource_pet/pet_resource_gen.go:2088:6: NewNameValueMust redeclared in this block
        internal/resource_pet/pet_resource_gen.go:770:6: other declaration of NewNameValueMust
internal/resource_pet/pet_resource_gen.go:2155:6: NameValue redeclared in this block
        internal/resource_pet/pet_resource_gen.go:837:6: other declaration of NameValue
internal/resource_pet/pet_resource_gen.go:2700:6: IdType redeclared in this block
        internal/resource_pet/pet_resource_gen.go:1539:6: other declaration of IdType
internal/resource_pet/pet_resource_gen.go:2770:6: NewIdValueNull redeclared in this block
        internal/resource_pet/pet_resource_gen.go:1609:6: other declaration of NewIdValueNull
internal/resource_pet/pet_resource_gen.go:2776:6: NewIdValueUnknown redeclared in this block
        internal/resource_pet/pet_resource_gen.go:1615:6: other declaration of NewIdValueUnknown
internal/resource_pet/pet_resource_gen.go:2782:6: NewIdValue redeclared in this block
        internal/resource_pet/pet_resource_gen.go:1621:6: other declaration of NewIdValue
internal/resource_pet/pet_resource_gen.go:2782:6: too many errors

I went through and read through a bit of the openapi json file and modified instances of categorys to add prefixes to their names but it got old after a little bit and I found your post here.

I also added:

      ignores:
        - id
        - category
        - category.id
        - tags
        - tags.name

to the generator_config.yml from this page but no success.

I pretty much followed the workflow step by step, copy pasting. I can confirm that the initial provider was configured and the plan ran as expected. Can you advise on what I’m potentially missing or if there is some additional undocumented step/something that has since changed?

9

Hey there @Arequ :wave:,

I created an issue to keep your discussion point going over here: Petstore workflow example from code-gen documentation is generating incorrect schema · Issue #152 · hashicorp/terraform-plugin-codegen-openapi · GitHub

Nothing that you did wrong, it looks like the OpenAPI spec file we download from Swagger Petstore in the tutorial has been updated with some recursive schemas. If you want to subscribe to that issue I linked you can track the progress of that getting fixed. (I’ve also linked the downstream Swagger issue there, but will link here as well: petstore 1.0.18 has wrong scheme! · Issue #125 · swagger-api/swagger-petstore · GitHub )