How can I automate creating variables.tf, outputs.tf, and .tfvars data? Or what is an efficient workflow?

Terraform
1

As my infrastructure grows, I’m hitting some bottlenecks, particularly having to do with managing variables, outputs, and module arguments.

Scenario:

Let’s say I’m creating the following resource.

resource “azurerm_resource_group” “default” {
name = var.default_resource_group_name
location = var.default_location
}

Next I have to manually open up variables.tf and define variables{} blocks for those 2 variables. I then have to open up outputs.tf and create output blocks for those variables (if I want them output). If I want to add default values, then I have to open up yet another file, terraform.tfvars, and add this same data yet again.

Having to repeat manually entering the same data into 3 different places, well, either I’m doing it wrong or this is woefully inefficient. All the time I saved managing infrastructure, I now spend editing my 4 different files. If I add a module to the mix, I now have yet another layer of arguments and variables to manage.

Terraform is smart enough to know that if I provide that block above, validate is going to explicitly tell me that I need a variables default_location {} block. If it can do that, why can’t it just create it for me? I feel like there should be a command option like --generate-variables, --generate-outputs etc.

As my infrastructure grows having to validate, open file, manually add missing variable, validate again… I understand why for the workflow init, validate, plan, but some resources have over 250 arguments by themselves. Doing that workflow manually is unsustainable and impossible to use in CI/CD pipelines.

Is there a better way? What automation options or tools are there that I may not be aware of, or what’s wrong with my workflow that using Terraform is now costing me more time to manage input and output files than I did before had I just edited the resource without terraform, especially when working in a team setting?

2

Hi @blue928,

If you are just systematically exposing all of a resource type’s arguments and attributes as module input variables and output values then my main suggestion would be to discard the module and have configurations that were using it just declare a resource directly instead.

A module is typically only useful if it provides something additional, such as its own values for at least some of the arguments, or a particular opinionated collection of several resource types. In that case, the job of the module author is to design the smallest interface that creates the intended abstraction, which requires human judgement that Terraform cannot automate.

Obviously I cannot see what you have written and so can’t give concrete suggestions, but my broad advice to start would be to avoid creating a module until there’s a clear need to create an abstraction. Modules add additional maintenance cost and so you’ll want to make sure their benefit outweighs that cost before creating one. From what you’ve described, it seems like you have at least a few modules whose cost is outweighing its benefit, which for me typically calls for removing that module and using the resources inside it inline instead.

3

I read this and watched the fantastic video that is spot-on when it comes to Terraform’s evolution in an organization. My organization is somewhere between late-stage pattern 3 and early 5. As we decompose our “terralith” we have inconsistencies among teams (patterns, naming conventions, variable and argument choices etc). These are starting to cause errors in CI/CD forcing a ticket-review process that is slowing things down.

Since a lot of my question seem to be about workflow, I’ll save that for this separate question. But I am still interested in some of the automation and programmatic aspects.

All resources have required and optional arguments. But in my organization, we have, for example, additional optional arguments that are required for us. Dev A in Japan creates a resource, forgets a couple of them or names them something obscure, etc. Dev B in America is blocked until they can convene and discuss. Given time zones, ticket review, this one issue is now a week or more delayed.

I need to automate this and create exact consistency so that Dev A starts out with exactly what Dev B would start with or is expecting; and, what CI/CD tests are expecting - templating the initial process, if you will.

Here are thoughts on how to achieve this:

  • Use Golang to autogenerate the files by querying the API
    ** How can I query the API to get a list of all required arguments for a specific resource?

I found that I can query for provider information, but I can’t find info to retrieve resource information. My thinking is when a developer wants to create a new resource, He’ll run a go or typescript to generate the manifest files along with expected naming conventions, and populate main.tf, variables.tf, outputs.tf, etc, with exactly what data that everyone is expecting. I’m looking form something like curl registry.terraform.io/providers/hashicorp/azurerm/v2.99/resource_group?required=yes This should show me all required arguments along with descriptions and other info I can use straight from the API.

  • Use CDKTF to generate an HCL manifest.tf file from JSON
    ** How can I use CDKTF to generate an HCL file?
    CDKTF is EXACTLY what I’m looking for - almost. HCL is seamlessly compatible with JSON. Running cdktf synth creates ./out/cdk.tf.out I’m so close! How do I turn that file into main.tf?!?

I know that I can use that file as a drop in replacement, but I don’t want my team members to have to learn typescript or json. If I can create a templatized JSON file containing exactly what I’m expecting users to start with, and if they can run some command like cdktf conver-to HCL main.tf then I’ve accomplished my goal.

If cdktf can create a manifest file and turn it into JSON, can’t it do the exact opposite?