Terraform test does not fail when it should

Terraform
1

I’m not sure if im doing something wrong here but i think terraform test should be failing but im getting all tests pass.

resource "test_assertions" "expect_failure" {
  component = "tags"

  check "fail" {
    description = "this test should fail"
    condition   = true == false
  }
  equal "scheme" {
    description = "should fail"
    got         = true
    want        = false 
  }
}

$ tf -v
Terraform v1.1.0-rc1

same outcome on 1.0.8

2

Hi @drewmullen1!

What you’ve described here does indeed seem like a bug. My guess, not yet verified, is that because the assertion condition here is entirely constant Terraform is calling the data resource too early (not during the apply step) and then not checking whether it succeeded at that step.

In “real” use of this the assertion data source configuration typically refers to a resource result not known until apply and so Terraform defers checking the assertion until the apply step.

If that’s true then it’s definitely a bug to be fixed, but if you are trying this only as a precursor to testing something derived from a resource result then you will hopefully find that if you include that resource result in the condition then you’ll get an error as expected.

3

:wave: So I actually distilled this down to this simple example because I got suspicious that terraform test wasn’t actually running my tests.

In this branch (another test suite entirely) I expect the terraform test to take about 45m (the resources take a long time to destroy). When the tests came back about 3m later i started experimenting; It seems like the resources arent being created and the tests arent being validated atm

apologies i am showing you 2 different test suites but the terraform-aws-label was just a much simpler test and was easier to distill down.

1 Like
4

Hi @drewmullen1,

Would you mind opening an issue in the Terraform GitHub repository to describe what you’ve observed, and if possible an example of how we can reproduce it? It does sound like you’ve found a bug which I’d like to investigate more, but it’ll be easier to do that over in GitHub so it can go through the same process we use for other bugs. Thanks!

1 Like
5

Was there a resolution here or a GitHub issue to follow? I’m seeing something similar. I am testing a module, and the test was passing with an equal check like this:

  equal "sg_description" {
    description = "Security group should match"
    got         = module.test.security_group_description
    want        = "Security group managed by Terraform"
  }

So, just to confirm that I was doing it properly, I changed the want value:

  equal "sg_description" {
    description = "Security group should match"
    got         = module.test.security_group_description
    want        = "Foo"
  }

Test still passes.

So … maybe I’m doing something wrong? But I wanted to see where this discussion led, but there’s no GitHub Issues link. :wink:

6

Looks like it led here:

It’s not clear to me if I’m hitting the same problem. Is there a way to tell if my assertions are just not being checked because of a failure that isn’t reported?

7

Ok, so if I do TF_LOG=error terraform test, I can see a failed AWS call. This is just a rough edge on terraform test for now that I can’t really do much about, from the sounds of things?

So for now ‘success’ either means “Test passed, or something failed before an assertion was checked”?