Hi All,
Am looking for an effective way to generate terraform code in HCL format. So, instead of writing Terraform code in HCL, am looking for a way to programatically generate HCL code. Can someone please point me in the direction that’ll help me acheive this ? Thanks.
Can you give some idea of what you want to generate it from? IE, do you want to convert a CloudFormation template, an ARM template or similar? Do you want to generate it from an existing set of cloud resources? Do you want a different DSL?
What is the motivation for wanting to generate HCL instead of writing it?
@brucellino1 , I am working on a project, where in dynamic terraform code can be generated (HCL or JSON format).
for eg, if someone wants to deploy an EKS cluster - then instead of writing the tf code, user only needs to input basic values - like cluster-name, region, etc and the tool will auto-generate tf code based on the user input.
Ah ok, you’re working on a sort of frontend to making templates, it sounds like. This seems to be a valid use case (not sure how common). It sounds like you would need to include the hcl library into whatever you are using as the frontend. Combining this with the go templating tools you can probably do what you want with minimal work – but you would be tied to the Go language.
However, I’m having a hard time understanding how templating out HCL is better than passing variables to a module…
This is typically what user’s have to do – someone writes the module, then publishes it and whoever uses it reads the module docs and passes their desired variables.
Can you explain what you need to do that cannot be done with passing variables via var or var-file ?
In general, a Terraform module (even the root module) should do what’s written on the box – it’s a declarative language.
If you’re looking to generate definitions by dynamically interpreting the user’s input, then perhaps the CDK is more adapted to you’re usage model?
Yes, its something like a front-end which can generate tf code in either JSON or HCL.
Also, is there any REST API’s that can fetch data (resource usage) from any terraform registry ? Like for eg, if i want to create and EKS cluster - i’d go to this site and check on the usage of that module, and write my own code - which is manual . So i want to automate this , by seeking user’s input and feeding it into the eks module and generating/executing code on user’s behalf.
i couldn’t anything helpful on this, can you please point me in the right direction ?
Anything is fine :
@brucellino1 thanks in advance
Thanks for these clarifications, I’m starting to see where you’re coming from 
So, let’s start by saying that generating arbitrary Terraform may not be a realistic or desirable goal – someone is always going to write the actual Terraform modules. However, generating some Terraform which _consumes existing Terraform modules may be realistic.
(also, bear in mind that whatever comes next here is just my opinion, and I’m by no means an authority on the topic!)
Let’s say you have a backend application, which:
Then you have a frontend which uses this backend to present a simpler interface to the user that says, let’s say:
Then you can generate a main.tf which uses those modules, using the HCL library, or even just a simple templating library. You can do this because the modules express very clearly what their input and output variables are, what their defaults are, etc.
However, I don’t think this could cover any but the very simplest of use cases… and it sounds very similar to what a Backstage catalogue would do.
Just curious - is this more or less what you had in mind?
Yes it does.
Yes am looking a way to create such backend, which can talk to any provider registry , fetch some modules - and with my code’s logic , create a meaningful and executable tf code.
can you tell me if there is a way to talk to the provider’s registry and achieve this ?
well, its different from what i have on my mind on this usecase
I would start with the registry API – get the source code of the module you want to use in the templating, and parse the variables.tf file which is inevitably in there. Then, use that to generate the user interface… from the input to the user interface you can write out (with the same library you used to parse the terraform files in the module) the main.tf which consumes it.
Again - I think the trick here is in the proper filtering of which modules can be used, because there are probably going to be modules designed for a specific purpose rather than truly general purpose ones, and you’re not guaranteed to be able to have arbitrary modules work together. But you can present a catalogue of known good combinations with appropriate filtering.
Sounds like fun! be sure to let us know if you get anything working!
This project sounds interesting!
Everything above seems like great discussion so far! I just have a few extra notes to keep in mind:
The Terraform Registry API is indeed a handy way to get access to a summary of the input variables and output values of a module without first downloading the module. Specifically, I think Latest Version for a Specific Module Provider is a good operation for that.
The thing I want to note is that this particular part of the API is specific to Terraform Registry (registry.terraform.io) and isn’t part of the general module registry protocol that third-party registries must implement. Therefore if you use this approach then your solution will be limited only to modules shared in the main public registry. I’m not sure if this is actually important for your goal so I wanted to mention it to be sure.
(If you want to work with arbitrary Terraform modules that might be in other registries or not listed in a registry at all then the only general answer is to install the module using terraform init -from-module=... and analyze what it installs using terraform-config-inspect. This is approximately what Terraform Registry itself is doing for each new module version published in order to obtain the metadata it exposes via its API.)
If the generated Terraform language code is intended only for automated use and not for ongoing maintenance by human developers, I’d recommend using the JSON variant of the Terraform language just because it’s often easier to generate correctly using JSON encoding libraries available in various languages.
If you are intending to generate a starting point for later human maintenance then it may be better to generate the native syntax, because that’s friendlier to human maintainers. You can use the hclwrite package to generate HCL grammar programmatically, and there’s an example in the package of generating an entirely new document from empty.
It sounds like you’ll be primarily interested in generating a module block, but you will probably also need to generate one or more provider blocks and possibly also a terraform block with a backend configuration in order to produce a self-contained Terraform root module. Shared modules typically expect to be passed in provider configurations from their caller, rather than declaring their own.
Thanks alot for all information @apparentlymart @brucellino1.
I tried downloading a module for gcp vm, but it didnt download anything.
curl -i https://registry.terraform.io/v1/modules/terraform-google-modules/vm/google/download 
HTTP/2 302
content-type: text/html; charset=utf-8
location: /v1/modules/terraform-google-modules/vm/google/7.8.0/download
server: terraform-registry/c5a98495e7cac9bf95029bb58bf3830b6154a83b
strict-transport-security: max-age=31536000; includeSubDomains; preload
accept-ranges: bytes
via: 1.1 varnish, 1.1 varnish
date: Mon, 12 Sep 2022 18:26:51 GMT
x-served-by: cache-iad-kjyo7100033-IAD, cache-hyd1100035-HYD
x-cache: MISS, MISS
x-cache-hits: 0, 0
vary: Accept-Encoding
content-length: 84
< href=“/v1/modules/terraform-google-modules/vm/google/7.8.0/download”>Found
Also i wanted to check, if there’s a way to interact with the provider to fetch source, instead of module
Hi @saad-uddin,
That response is an HTTP redirect telling you to repeat your request at a different URL, specified in the Location header.
curl doesn’t automatically follow redirects, so to practice with that command you’ll need to run curl again with the new URL to see what the new request returns:
$ curl -i https://registry.terraform.io/v1/modules/terraform-google-modules/vm/google/7.8.0/download
HTTP/2 204
cache-control: public, max-age=604800, stale-if-error=31536000
last-modified: Thu, 23 Jun 2022 03:01:32 GMT
server: terraform-registry/c5a98495e7cac9bf95029bb58bf3830b6154a83b
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-terraform-get: git::https://github.com/terraform-google-modules/terraform-google-vm?ref=v7.8.0
via: 1.1 varnish, 1.1 varnish
accept-ranges: bytes
date: Mon, 12 Sep 2022 18:39:04 GMT
age: 1946
x-served-by: cache-iad-kjyo7100141-IAD, cache-pdx12328-PDX
x-cache: HIT, HIT
x-cache-hits: 1, 1
vary: Accept-Encoding, X-Terraform-Version
The Download Source Code for a Specific Module Version operation (which is what we called here) is defined to return a 204 No Content response with a header X-Terraform-Get describing the real download location.
That matches the response above, and the X-Terraform-Get header contains the following string: git::https://github.com/terraform-google-modules/terraform-google-vm?ref=v7.8.0
The challenge at this point is that this is a raw module source string and there is not a well-defined specification for how to interpret these. Instead, the specification is just whatever Terraform CLI itself does. This is why I described using terraform init -from-module=... to download the module instead; that command knows how to understand the above string and retrieve the module source code using Git.
$ terraform init -backend=false -from-module=git::https://github.com/terraform-google-modules/terraform-google-vm?ref=v7.8.0
Copying configuration from "git::https://github.com/terraform-google-modules/terraform-google-vm?ref=v7.8.0"...
Terraform initialized in an empty directory!
The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
This particular module package is interesting because the root directory doesn’t contain a Terraform module, and so Terraform treated it as an empty directory. It isn’t really fully empty, but from Terraform’s perspective it doesn’t include any .tf files and so there’s nothing in that directory that Terraform itself knows how to read.
However, despite this weird error message there are still files in the current working directory afterwards:
$ ls
autogen autogen_modules.json build CHANGELOG.md
CONTRIBUTING.md docs examples LICENSE Makefile
modules README.md test
The modules subdirectory in particular contains what the Terraform Registry web UI describes as “sub-modules”, which are also Terraform modules you can analyze.
(Note: I used -backend=false on the terraform init command above to prevent any backend initialization. It’s not valid for a shared Terraform module to include a backend configuration, but this particular command is asking Terraform to treat a shared module as if it were a root module and so a malicious module package could potentially include a backend configuration that Terraform would then try to initialize, potentially sending any ambient credentials on your system to an attacker-provided location. Disabling the backend initialization prevents that from happening.)
If you are willing to support only the official public Terraform Registry then there is an easier path in retrieving “Latest Version for a Specific Module Provider” in the public-registry-specific API:
curl https://registry.terraform.io/v1/modules/terraform-google-modules/vm/google
In this response you should see for the main module (which doesn’t exist in this package) and for each submodule the JSON properties inputs and outputs, which will directly describe which input variables the module expects and which output values it produces.
The registry doesn’t currently expose exactly which provider configurations the module expects, so you’d need to infer that separately somehow today. For example, you could in principle assume that because the last part of the module’s address is “google”, representing Google Cloud Platform, that the module will need a default (unaliased) configuration for the hashicorp/google provider. It’s possible that a module might require additional (aliased) provider configurations too; that currently is not exposed in the API, so you’d need to do direct analysis to learn that.
Hi @apparentlymart Thanks for the details. However, I am starting to see that this really isn’t very helpful. is there a way to fetch all the supported “resource” code from terraform provider/registry? cause if I can fetch the resource, then I can write my own code to create modules dynamically.
eg resource code : GCP_COMPUTE_ENGINE
If we can fetch a resource along with entire supported values (both required and optional) , then it will be easy to progress further.
your thoughts on this ?
Hi @saad-uddin,
Provider plugins are executable files rather than data so there unfortunately isn’t a comparable data-parsing-only approach to reading provider schemas.
However, there is a path to get there for a specific provider by running Terraform CLI as follows:
Generate a file containing just a requirement on the provider you are interested in. For example:
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "4.36.0"
}
}
}
In the directory containing that file, run terraform init to make Terraform install that provider into the local provider plugin cache.
Run terraform providers schema -json, which will cause Terraform CLI to start up the provider plugin, request its schema information, and report the result as JSON.
You can then capture the output of the last command and consume it from your own program.
I would recommend caution when doing this because providers are arbitrary executable code which will run on your computer when you run the last command. It is potentially possible for a provider to take malicious actions when Terraform CLI starts it up to ask for schema. Therefore you should make sure to do this only for plugins you have some reason to trust, rather than arbitrary provider addresses specified by an external user.
Thanks @apparentlymart , this helps. I was able to fetch the schema and isolate required resources. I wondering if you could help me find a way to generate HCL code based on any json-schema input ?
Hi @saad-uddin,
This part of my earlier comment is my general advice for how to get started here:
If the generated Terraform language code is intended only for automated use and not for ongoing maintenance by human developers, I’d recommend using the JSON variant of the Terraform language just because it’s often easier to generate correctly using JSON encoding libraries available in various languages.
If you are intending to generate a starting point for later human maintenance then it may be better to generate the native syntax, because that’s friendlier to human maintainers. You can use the
hclwritepackage to generate HCL grammar programmatically, and there’s an example in the package of generating an entirely new document from empty.
I can’t really be more specific while we’re talking in such general terms, but if you try one of the options above and have more specific questions based on what you learn then I’d be happy to try to help with those!
Hi @apparentlymart thanks for the help. I was able to generate some tf code from the given schema, however i need one small help again. in this documentation, hashicorp has organized all the GCP resources , where in each resource type has its relevant “resources” and “data” sections : eg Terraform Registry
Can you tell me if there’s a way to fetch all this data form a backend ? I need to map each service of a provider with its “resources” and “data” blocks , from the providers schema, and since schema doesn’t have that mapping feature, but the above documention does, hence i’d like to fetch that list from a backend ?
is it possible ? if so, can you please help me with an eg ?
Hi @saad-uddin!
It seems like you are referring to the documentation index. That isn’t really data in a sense that is useful for external integration since it’s just a directory of Markdown files in the provider repository.
The set of managed resource types and data source types in a provider is included in the JSON schema output, though… indeed, for most providers that represents most of the content of the schema. If you can say some more about why you can’t get the information you need from the schema JSON I may be able to help with that.