Request for Testing: Terraform Test!

Terraform
1

Hey all!

The Terraform team is thrilled to present Terraform Test. This is in the alpha release of 1.6, so it is not ready for production.

We would love for folks to give it a spin and provide feedback :smiley:.

You can download it via the releases website or on GitHub (see changelog).

The documentation for the new testing framework is still a work-in-progress, but it can be found here.

Please respond to this topic or email me at oismail@hashicorp.com for feedback.

Thank you!

2

Hi @omarismail

Thanks for your efforts in this alpha release!

My understanding from reading your docs is that the current implementation tests assertions on an applied plan - integration tests. This requires resource creation, which can take some time depending on the cloud resource in use.

My own experience is that this testing is beneficial, however there is a faster dev loop made possible by testing the plan.

Plan testing obviously still requires init, etc. but since no resources are deployed it makes it possible to more quickly test, e.g. local expressions and conditions.

I have done quite a lot with Terratest and I have written a wrapper here to make the syntax more like written English: Azure/terratest-terraform-fluent: Go module for Terraform module testing with fluent assertions (github.com). I have implemented this library on this module and it has really helped increase quality.

Would love to know if you plan to introduce assertions on a plan & happy to connect and discuss in more detail :slight_smile:

3

Hey Matt,

This test framework includes plan only testing! I know the docs might be hard to read, but check out this section

1 Like
4

My apologies, Iā€™ll give it a whirl

5

Hi all,

Im trying to test the new alpha build for the terraform test v2 function. For some reason this command is not finding and executing my test. Here is a WIP pr to highlight the code im running against: WIP: Prepatory PR to swap terratest with `terraform test` by drewmullen Ā· Pull Request #127 Ā· aws-ia/terraform-aws-vpc Ā· GitHub

To recreate:

$ git clone git@github.com:drewmullen/terraform-aws-vpc-1.git
$ cd terraform-aws-vpc-1
$ git checkout terraform-test-native
$ terraform test                                      

Success! 0 passed, 0 failed.


$ terraform test -test-directory=./tests/basic        

Success! 0 passed, 0 failed.

seems to just not find the testsā€¦ but im just trying to recreate what i find:

7

I changed my test above to use command = plan and try just validating a locals value. same thingā€¦ doesnt seem to see the test at all

terraform -version                          
Terraform v1.6.0-alpha20230719
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.9.0
+ provider registry.terraform.io/hashicorp/awscc v0.56.0
8

Hi @dmullen, thanks for your feedback.

tl;dr: I think you need to cd tests/basic first, and then run terraform test without the -test-directory flag.

The terraform test command discovers tests based on the current directory, and assumes the current directory contains the configuration you want to test.

In your case, I think you are calling terraform test from the root directory of the repository, which contains the main configuration for the module, but looking at the PR youā€™ve created you have actually defined additional configuration files within the subdirectory. To test those new configuration files you want to change into the directory that contains them and execute terraform test from there.

Having said that, I was surprised terraform test -test-directory=./tests/basic didnā€™t at least discover the test files. I think youā€™ve discovered a bug here as I can replicate that behaviour. The test framework isnā€™t discovering test files when your test directory is nested by more than one subdirectory. Iā€™ve filed an issue for this and weā€™ll get it fixed for the next prerelease.

A workaround here is just to make sure your .tftest files are in a direct subdirectory of your configuration files.

Thanks for the feedback and bug discovery!

1 Like
9

Hi @liamcervante thanks for helping me look into this.

It looks like you were right:

$ cd tests/basic 

$ ../../terraform test
basic.tftest... pass
  run "validate_locals"... pass

Success! 1 passed, 0 failed.

Basically i was just trying to replicate what i see in the core repo testdata dir. Basically just playing with the command to see how we might want to use it to replace terratest in our modules.

I found another :smiley: `terraform test` crash when using `module.source` Ā· Issue #33586 Ā· hashicorp/terraform Ā· GitHub

no complaints here though - i cannot wait for this feature to replace terratest :tada:

10

I am currently evaluating tools to test the infrastructure which was applied by Terraform. Therefore I like the approach of the built-in feature ā€œModule Testingā€ of Terraform. I am hesitant to currently use it in production because it is still experimental, and considering to use Terratest in the meantime. Is there a roadmap when ā€œModule Testingā€ is fully supported and production ready?

11

Hey @mymichu ,

Is there a roadmap when ā€œModule Testingā€ is fully supported and production ready?

Yes! This current feature is in the alpha release of 1.6, and we intend to make it Generally Available for the official release of 1.6, which will be around the fall (September/October) time frame.

1 Like
12

I got some tests working yesterday and Iā€™m generally pleased with the direction this is headed. My primary ask for the 1.6 release is that terraform fmt needs to also support .tftest files!

Iā€™d love a way to catch warnings in tests as well. Maybe some kind of strict option in the run block to make it fail-on-warning?

1 Like
13

Hi @threemachines, glad to hear itā€™s meeting expectations!

The good news is that weā€™ll definitely be supporting the test files in terraform fmt, that feature will be available in the next alpha release. The change has already been merged here.

Maybe some kind of strict option in the run block to make it fail-on-warning?

This is an interesting idea, and something that should be possible. Iā€™ll make sure the product team are made aware of it and we can see about prioritisation.

Thanks so much for the feedback!

15

Hi everyone! Weā€™ve just released a new alpha version for v1.6: Release v1.6.0-alpha20230802 Ā· hashicorp/terraform Ā· GitHub

For terraform test specifically the new version contains many new features and bug fixes discussed in this thread and within several Github issues.

One important change in this new alpha is the file extension for testing files has been updated to .tftest.hcl.

Many thanks!

1 Like
16

Hi all! Weā€™ve released a new alpha version in the 1.6 series: https://github.com/hashicorp/terraform/releases/tag/v1.6.0-alpha20230816.

It includes further refinements to terraform test, along with improved behavior for get and init -from-module. Thanks!

17

Hey all - I got a weird crash when having more than one test in a tftest.hcl file. Using beta-1.

I logged it here: terraform test crash on multiple run blocks in file Ā· Issue #33780 Ā· hashicorp/terraform (github.com)

If you want to see the actual code then I have it in a branch here:
terraform-azurerm-lz-vending/modules/subscription/unit.tftest.hcl at spike/terraformtest Ā· Azure/terraform-azurerm-lz-vending (github.com)

18

Hi all,

Are there any plans for an idempotency test?

In terratest there is the ability to ensure that a plan following an apply step does not result in any planned changes.

Proposed implementation:

run {
  command = "apply_idempotent"
  # ...
}
19

Hi @matt-FFFFFF, thanks for the suggestion!

We do have plans for the introduction of additional functions within test files (for example, returning the create/update/delete status) of a resource from a plan. Iā€™ve made made a note of your suggestion as something to explore along with these.

Thanks again!

20

My understanding from reading your docs is that the current implementation tests assertions on an applied plan - integration tests. This requires resource creation, which can take some time depending on the cloud resource in use.

What value would there be in a testing framework that doesnā€™t actually test based on resource creation? The terraform plan is already a ā€œkind of testā€ which is actually what created a lot of friction for having formal testing; everyone would argue that you donā€™t need testing because of the plan. That was always a bad idea because what you want to test is what actually happens. I have had instances where details of what I deployed didnā€™t matchā€¦ for instance when creating an Azure App Service Plan, I have had it deploy a Windows-based platform when I requested a Linux-based platform. You can only find that when deploying infrastructure.

The notion that it takes time is kind of silly. You automate the process and let it go. It takes a lot more time and other expense to troubleshoot something in production that could have been caught with a simple test.

21

Hi @liamcervante

Some further feedback.

I have a module with a complex input variable, a map(object({...})).

This variable uses optional() to provide default values for some of the properties.

Terraform test does not seem to honour the optional functions and I therefore have to declare the entire object value inside the test. This copy/paste is error prone.

Request: Is it possible to have the test function read the optional defaults?

Thanks!

Here is the input variable for reference:

variable "virtual_networks" {
  type = map(object({
    name                = string
    address_space       = list(string)
    resource_group_name = string

    location = optional(string, "")

    dns_servers = optional(list(string), [])

    ddos_protection_enabled = optional(bool, false)
    ddos_protection_plan_id = optional(string, "")

    hub_network_resource_id         = optional(string, "")
    hub_peering_enabled             = optional(bool, false)
    hub_peering_name_tohub          = optional(string, "")
    hub_peering_name_fromhub        = optional(string, "")
    hub_peering_use_remote_gateways = optional(bool, true)

    mesh_peering_enabled                 = optional(bool, false)
    mesh_peering_allow_forwarded_traffic = optional(bool, false)

    resource_group_creation_enabled = optional(bool, true)
    resource_group_lock_enabled     = optional(bool, true)
    resource_group_lock_name        = optional(string, "")
    resource_group_tags             = optional(map(string), {})

    vwan_associated_routetable_resource_id   = optional(string, "")
    vwan_connection_enabled                  = optional(bool, false)
    vwan_connection_name                     = optional(string, "")
    vwan_hub_resource_id                     = optional(string, "")
    vwan_propagated_routetables_labels       = optional(list(string), [])
    vwan_propagated_routetables_resource_ids = optional(list(string), [])
    vwan_security_configuration = optional(object({
      secure_internet_traffic = optional(bool, false)
      secure_private_traffic  = optional(bool, false)
      routing_intent_enabled  = optional(bool, false)
    }), {})

    tags = optional(map(string), {})
  }))
22

Hi @matt-FFFFFF, thanks for this! Definitely a bug, you should be able to do that. Iā€™ve filed this into the repo with `terraform test` is not applying defaults to variables referenced in assertions Ā· Issue #33864 Ā· hashicorp/terraform Ā· GitHub, it also has a simplified example.

Will try and get this fixed for beta2.

Thanks again! Good find!

1 Like