What you’ve described here does indeed seem like a bug. My guess, not yet verified, is that because the assertion condition here is entirely constant Terraform is calling the data resource too early (not during the apply step) and then not checking whether it succeeded at that step.
In “real” use of this the assertion data source configuration typically refers to a resource result not known until apply and so Terraform defers checking the assertion until the apply step.
If that’s true then it’s definitely a bug to be fixed, but if you are trying this only as a precursor to testing something derived from a resource result then you will hopefully find that if you include that resource result in the condition then you’ll get an error as expected.
So I actually distilled this down to this simple example because I got suspicious that terraform test wasn’t actually running my tests.
In this branch (another test suite entirely) I expect the terraform test to take about 45m (the resources take a long time to destroy). When the tests came back about 3m later i started experimenting; It seems like the resources arent being created and the tests arent being validated atm
apologies i am showing you 2 different test suites but the terraform-aws-label was just a much simpler test and was easier to distill down.
Would you mind opening an issue in the Terraform GitHub repository to describe what you’ve observed, and if possible an example of how we can reproduce it? It does sound like you’ve found a bug which I’d like to investigate more, but it’ll be easier to do that over in GitHub so it can go through the same process we use for other bugs. Thanks!