Is it possible to "render" a Terraform configuration?

Some context: I am working on a tool to benchmark the refresh performance of each resource type in a Terraform workspace. To do this I create a temp dir, copy the state file over with just the resources I want to measure. Also I copy over the provider, variable and terraform.required_providers configuration. Then measure the time to run terraform refresh, Terraform will only refresh those resources I put into the state file. Finally I rank the results and generate a nice report so you can see what resources are slowing down the refresh time the most.

The issue is when someone defines their provider configuration with a value from another resource/data_source, e.g:

provider "aviatrix" {
  username                = data.aws_ssm_parameter.avx_uname.value
  password                = data.aws_ssm_parameter.avx_pword.value
  controller_ip           = data.aws_ssm_parameter.avx_ctrl_ip.value
}

Then my technique breaks down because Terraform will complain that the data/resource is not defined in the root module. If I do copy over the data/resource configuration into the temp tf configuration, then the refresh times are no longer accurate because terraform refresh will try to refresh the data sources as well instead of just the resources in the state file. This is not the end of the world since it still allows me to make an apples to apples comparison of refresh times, but the numbers are not absolutely accurate.

My question then is: Can I “render” the terraform configuration and then copy over the provider configuration with hard-coded values instead of the data source interpolation expressions. Basically going from the provider configuration above to:

provider "aviatrix" {
  username                = "uname"
  password                = "pword"
  controller_ip           = "1.2.3.4"
}

Paging TF wizard @apparentlymart as this may be an interesting question for you.

After more thought I think it may be possible to utilize terraform console to “render” the expressions within the provider block. Though to do this with sensitive values I need to use the nonsensitive function, which brings the min tf version to 0.15

Also I could do something more messy with trying to parse the expression myself then looking up the value in the state, but that seems fragile.

Hi @CyrusJavan,

Sorry for not replying sooner. I’ve been taking some time away from work.

Unfortunately my short answer is that there is no direct way to get the result you’re looking for, because Terraform’s normal mode of operation is to resolve these values dynamically based on the plan in progress, rather than simply to plug in the existing values in state (which I assume is what you were hoping to do here).

Thinking more about your stated goal than your current strategy, I have an alternative strategy to suggest:

In recent versions of Terraform we included a streaming JSON output mode for terraform plan and terraform apply, which was originally intended to allow different UI layers wrapping Terraform but could also potentially serve as a (admittedly rather coarse) means to measure how long various different Terraform actions are taking.

If you run terraform plan -refresh-only -json then Terraform should create a refresh-only plan while producing on stdout a series of JSON objects that report the sequence of actions it’s taking to do so.

In your case, you could monitor for the refresh_start and refresh_complete messages in particular, correlate them by hook.resource.resource, and then use the difference in @timestamp to get a somewhat-accurate elapsed time for each refreshed resource instance.

Terraform would still do the various other work involved in creating a plan, but you could ignore those messages and thus gather only the data you care about. (Might be interesting to experiment with running with and without --parallelism=1 to see if concurrent non-refresh actions have any material impact on how long a refresh action takes; I’d expect it would be immaterial because refreshing is I/O-bound, but I’ve never actually measured it!)

An interesting thing about this approach is that in principle you could instrument your routine Terraform runs in this way, both generating the plan and gathering the timing information you want. The trick there would be that Terraform’s designed to either produce JSON output or human-readable output, not both, and so to do this would mean you’d lose the typical human-readable log output. You could at least still get the main plan diff part though:

terraform plan -refresh-only -json -out=tfplan
terraform show tfplan

@apparentlymart Thank you for the thoughtful and detailed reply as always! :smiley:

Today I learned about the -json flag for plan/apply! That is pretty neat and now it makes more sense to me how Terraform cloud/enterprise is able to track progress of Terraform actions.

I am already three quarters down the road with my initial implementation so I will have to play with that idea another time. I ended up creating an evaluate function like so func evaluate(attr *hclwrite.Attribute, varFile string) cty.Value that uses the console to evaluate the given expression, then I set the attribute value to the returned cty.Value. Then I guard this evaluation behind a check for TF version 0.15 since I need to use the nonsensitive function. (It surprises me still how many 0.12 workspaces I come across in my own organization and beyond).

P.S. From preliminary results it seems azurerm is the slowest turtle in the race compared to aws and google.

@apparentlymart I tried out your suggested method and I think it is significantly more accurate than my original method! :smiley:

// New method with event log from `terraform plan -refresh-only -json`, times are an average time for a single resource of a type to refresh
Refresh Time for Whole Workspace: 19.196s
+---------------------------------------------------------------------------------+
| Individual Resource Type Refresh Statistics                                     |
+---------------------------------------------+-------+---------------------------+
| RESOURCE TYPE                               | COUNT | AVERAGE TIME PER RESOURCE |
+---------------------------------------------+-------+---------------------------+
| aviatrix_spoke_gateway                      |    10 |                    2.946s |
| aviatrix_aws_tgw                            |     1 |                    2.824s |
| aviatrix_transit_gateway                    |     3 |                    2.079s |
| aviatrix_vpc                                |     1 |                    1.695s |
| aws_vpc                                     |    11 |                    1.658s |
| aviatrix_transit_external_device_conn       |     2 |                     786ms |
| aviatrix_spoke_transit_attachment           |    10 |                     664ms |
| aviatrix_aws_tgw_transit_gateway_attachment |     1 |                     420ms |
| aws_internet_gateway                        |    11 |                     206ms |
| aws_subnet                                  |    40 |                     197ms |
| aws_vpc_ipv4_cidr_block_association         |    10 |                     196ms |
| aws_route_table                             |    20 |                     192ms |
| aws_route                                   |    10 |                     188ms |
| aws_route_table_association                 |    40 |                     185ms |
| azurerm_virtual_network                     |     1 |                     144ms |
| azurerm_subnet                              |     2 |                      87ms |
+---------------------------------------------+-------+---------------------------+
// Old method, times are to refresh all the resources of a certain type
Refresh Time for Whole Workspace: 17.95s
+----------------------------------------------------------------------------------------------+
| Individual Resource Type Refresh Statistics                                                  |
+---------------------------------------------+-------+----------------------------------------+
| RESOURCE TYPE                               | COUNT | AVERAGE REFRESH TIME OF 3 MEASUREMENTS |
+---------------------------------------------+-------+----------------------------------------+
| azurerm_subnet                              |     2 |                                10.564s |
| azurerm_virtual_network                     |     1 |                                10.383s |
| aviatrix_transit_gateway                    |     3 |                                 5.978s |
| aws_vpc                                     |    11 |                                  5.57s |
| aviatrix_spoke_gateway                      |    10 |                                 5.146s |
| aviatrix_aws_tgw                            |     1 |                                 4.066s |
| aws_route_table_association                 |    40 |                                 3.482s |
| aws_subnet                                  |    40 |                                 3.356s |
| aviatrix_transit_external_device_conn       |     2 |                                 2.994s |
| aviatrix_vpc                                |     1 |                                 2.768s |
| aws_route_table                             |    20 |                                 2.678s |
| aws_internet_gateway                        |    11 |                                 2.626s |
| aws_vpc_ipv4_cidr_block_association         |    10 |                                 2.462s |
| aws_route                                   |    10 |                                   2.4s |
| aviatrix_spoke_transit_attachment           |    10 |                                 2.293s |
| aviatrix_aws_tgw_transit_gateway_attachment |     1 |                                 1.503s |
| aws_availability_zones                      |    11 |                                  859ms |
+---------------------------------------------+-------+----------------------------------------+

azurerm had a crazy improvement. Maybe there is a long time to initialize the azurerm provider with the temporary dir method.

I also tested out parallelism=10 vs parallelism=1, and as expected there was not much difference in the large CSPs. However, in my own Aviatrix provider our aviatrix_spoke_gateway had a huge difference. I think I know why and perhaps I will be able to improve our performance. :grinning_face_with_smiling_eyes:

// parallelism=10
Refresh Time for Whole Workspace: 25.644s
+-------------------------------------------------------------------------------------------------+
| Individual Resource Type Refresh Statistics                                                     |
+---------------------------------------------+-------+---------------------------+---------------+
| Resource Type                               | Count | Average Time Per Resource | Average*Count |
+---------------------------------------------+-------+---------------------------+---------------+
| aviatrix_spoke_gateway                      |    10 |                    3.521s |       35.213s |
| aws_vpc                                     |    11 |                    1.736s |       19.091s |
| aws_subnet                                  |    40 |                     213ms |        8.502s |
| aws_route_table_association                 |    40 |                     196ms |        7.849s |
| aviatrix_transit_gateway                    |     3 |                    2.188s |        6.565s |
| aviatrix_spoke_transit_attachment           |    10 |                     574ms |        5.737s |
| aws_route_table                             |    20 |                     199ms |        3.977s |
| aviatrix_aws_tgw                            |     1 |                    3.281s |        3.281s |
| aws_internet_gateway                        |    11 |                     192ms |        2.108s |
| aws_vpc_ipv4_cidr_block_association         |    10 |                     186ms |         1.86s |
| aws_route                                   |    10 |                     183ms |         1.83s |
| aviatrix_transit_external_device_conn       |     2 |                     868ms |        1.737s |
| aviatrix_vpc                                |     1 |                    1.478s |        1.478s |
| azurerm_subnet                              |     2 |                     385ms |         770ms |
| aviatrix_aws_tgw_transit_gateway_attachment |     1 |                     193ms |         193ms |
| azurerm_virtual_network                     |     1 |                     146ms |         146ms |
+---------------------------------------------+-------+---------------------------+---------------+

// parallelism=1
Refresh Time for Whole Workspace: 1m33.744s
+-------------------------------------------------------------------------------------------------+
| Individual Resource Type Refresh Statistics                                                     |
+---------------------------------------------+-------+---------------------------+---------------+
| Resource Type                               | Count | Average Time Per Resource | Average*Count |
+---------------------------------------------+-------+---------------------------+---------------+
| aws_vpc                                     |    11 |                    1.905s |       20.952s |
| aviatrix_spoke_gateway                      |    10 |                    1.171s |       11.707s |
| aws_subnet                                  |    40 |                     216ms |        8.644s |
| aws_route_table_association                 |    40 |                     195ms |        7.799s |
| aviatrix_transit_gateway                    |     3 |                     2.09s |        6.269s |
| aws_route_table                             |    20 |                     227ms |        4.545s |
| aviatrix_aws_tgw                            |     1 |                      3.1s |          3.1s |
| aws_internet_gateway                        |    11 |                     201ms |        2.215s |
| aviatrix_transit_external_device_conn       |     2 |                    1.059s |        2.118s |
| aws_vpc_ipv4_cidr_block_association         |    10 |                     203ms |        2.031s |
| aws_route                                   |    10 |                     188ms |        1.883s |
| aviatrix_spoke_transit_attachment           |    10 |                     163ms |        1.634s |
| aviatrix_vpc                                |     1 |                    1.337s |        1.337s |
| azurerm_subnet                              |     2 |                     257ms |         514ms |
| azurerm_virtual_network                     |     1 |                     408ms |         408ms |
| aviatrix_aws_tgw_transit_gateway_attachment |     1 |                     187ms |         187ms |
+---------------------------------------------+-------+---------------------------+---------------+

Thanks again for the thoughtful reply, super helpful!