Request for Testing: Terraform Test!

Terraform
23

We just released Terraform v1.6.0-beta2!

Thanks everyone for your feedback so far, please keep it coming!

1 Like
24

what am i doing wrong here @liamcervante ? trying to use -filter

$ ./terraform test -filter=examples_ipam.tftest.hcl

Success! 0 passed, 0 failed.

$ ll tests/examples_ipam.tftest.hcl      
-rw-r--r-- 1 dm staff 213 Oct  3 14:03 tests/examples_ipam.tftest.hcl

$ cat -p tests/examples_ipam.tftest.hcl
run "ipam_setup" {
    command = apply
    module {
        source = "./tests/hcl_fixtures/ipam_base"
    }
}

run "validate_ipam_vpc" {
    command = apply
    module {
        source = "./examples/ipam"
    }
}

$ ./terraform -v   
Terraform v1.6.0-rc1
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v5.19.0
+ provider registry.terraform.io/hashicorp/awscc v0.61.0

$ ./terraform test                                 
tests/examples_ipam.tftest.hcl... in progress
...
25

Hi @dmullen, the filter flag accepts relative paths, so I think -filter=tests/examples_ipam.tftest.hcl should work. The argument as you have it would pick up a test file within the root configuration directory.

Iā€™ll see if I can update the docs to be clearer.

Thanks!

26

Hi, Iā€™m trying to get started with terraform test and have hit a stumbling block which I have posted about here: How do I pass provider information to tests?

Any help would be appreciated!

27

Hello,
is there a release ready documentation?

28

Hi @AZZ

Terraform v1.6.0 has now been released, with full documentation.

A good starting place would be here: Command: test | Terraform | HashiCorp Developer

We also have a Terraform language page on test files here: Tests - Configuration Language | Terraform | HashiCorp Developer

Hope these help!

29

@omarismail great timing with the terraform test-release - as Iā€™m just in the middle of my master thesis evaluating Terraform test approaches: Quantitative Cost Assessment of IaC Testing :sweat_smile:

On a side note - if someone has input on how to improve my tests / test cases, that would be highly appreciated: code repo :wink:

Regarding static testing (unit testing), I was able to cover

I was not able to cover the following static unit test with terraform test (so I used pytest instead) - input would be welcome :wink::

The missing DCs idempotency (dc5), service (dc7) and syntax (dc8) do not really make sense in the context of static unit tests - I think?

Using terraform test for dynamic testing, I cover the following cases (overlap with static tests is intended to compare them):

I was not able to cover idempotency checks with terraform test and had to use terratest instead: idempotency check

I did not come up with dynamic test cases for documentation (dc4), security (dc6), service (dc7) or syntax (dc8) - especially input for security or service related test cases would be highly welcome :thinking: :sweat_smile:

30

Hey @hashicorp20 , great to see this is helping your master thesis. Wanted to share that we will be adding mocking capabilities in the next release, so that may also give you more tools for your research!

If you wanted to chat, feel free to email me at oismail@hashicorp.com and we can setup a time :).

1 Like
31

[sorry to bump an old post, but wanted to comment on this]

This was always Hashicorpā€™s reason for not adding better unit testing / mock providers / etc. sooner.

But personally, Iā€™m so glad this functionality is here.

Terraform modules and code can be quite complex, so having a simple way to do testing on the plan and / or against a mock provider that will always succeed may not be sufficient, but it is necessary. A lot of the things that tend to break or be most fragile are things like data structures, use of HCL / Terraformā€™s odd and somewhat hard to read language features, so being able to quickly unit test lots of different scenarios or combination of variables adds a lot of value ā€“ even if it doesnā€™t prove that everything works end to end.

And yes, in some of these cases, the plan is the test, but having to write your own framework in a high level language to run / save the plan, and perform testing on that data structure is not ideal.

Worth reading this (old) issue if you havenā€™t already.

Also, there are certain resources that are difficult to integration test (because theyā€™re immutable or impossible to delete, letā€™s say), or projects where itā€™s not feasible security-wise to provide access to provision real cloud infrastructure. For example, I donā€™t think itā€™s possible today, but if you could mock (in the mock provider) the side effect of an upstream provider disallowing a specific value for an attribute, or even better, if big providers like AWS or GCP could provide a mock provider that would know the allowed / possible values and / or provide realistic mock values back, it might not be perfect, but it would allow a lot of things.

Long term, one could imagine even better mocking functionality for mock providers that would let you test certain types of side effects that might be difficult to test in normal types of tests, even integration tests.

32

Just want to say, got a chance to talk with @omarismail a while back, and was glad to have been able to provide some input into the way the test framework ended up working out.

While there are some things Iā€™m super happy about, as well as some things Iā€™d love to be different, Iā€™m really happy that some progress has finally been made here.

Iā€™ve finally started getting around to learning the new test framework this week, and am really happy with some of the things that are possible.